The AI glossary, in plain English

A small private file that holds secrets like API keys as NAME=value lines, kept out of your code and out of git.

Libraries such as python-dotenv read the file and put each value into the program's environment variables, where SDKs find them. The .env file must be listed in .gitignore before your first commit. Projects usually commit a `.env.example` with the names but no real values.

Like a locked drawer for the shop keys, separate from the recipe binder everyone can read.

A file listing the files and folders git should never track, such as .env and .venv.

Anything matching a pattern in .gitignore is left out of `git status` and `git add`. Use it for secrets, virtual environments, caches, large data and model files. Add `.env` before your very first commit, because history remembers anything committed.

Like a "do not photograph" list for the logbook: the safe's combination never goes in.

An open standard that lets agents from different organisations discover each other and work together on tasks.

Each agent publishes an agent card describing its skills and how to reach it. Requests create tasks with standard states, and results come back as artifacts. Agents stay opaque to each other, never sharing their internal tools or prompts. Google announced A2A in 2025 and donated it to the Linux Foundation.

Like the standard way businesses place orders with each other: a catalogue, an order number and a delivery note.

When an AI system deliberately declines to answer because its sources do not support an answer.

Good abstention means saying "I don't know" for questions the documents cannot answer while still answering the ones they can. Track both the correct abstention rate and the false abstention rate. Techniques include an exact abstention phrase in the prompt, a retrieval-score gate, cite-or-abstain checks and post-answer verification.

Like a good pharmacist who says "let me check with the doctor" instead of guessing a dose.

A summary written in new words that capture the meaning of the source.

This is what LLMs produce by default. It reads naturally, but it can drift from the source, blend separate facts or overstate trends. Asking for counts and quotes keeps it honest.

Like retelling a film to a friend in your own words.

A secret string that proves who you are to Hugging Face, so you can download gated models or pay for inference.

You create access tokens in your Hugging Face account settings. Read tokens can download; write tokens can also upload; fine-grained tokens can be limited to specific repos and actions. Libraries read the token from the HF_TOKEN environment variable or from hf auth login. Treat it like a password and never commit it to Git.

Like a library card: it lets the library know who is borrowing, and whoever holds it can borrow in your name.

A structured list of the buttons, links, fields and labels on a page that browsers produce for screen readers.

Browser agents can read it as text instead of looking at screenshots, choosing elements by label rather than by pixel coordinates. This is cheaper and more precise, and robust to cosmetic redesigns, but it fails on pages with poor labelling and misses anything that only appears visually.

Like a restaurant menu read aloud: you hear every dish and price without seeing the layout.

A simple non-linear function applied to each output of a layer so the network can learn curves, not just straight lines.

Without activations, any stack of linear layers collapses into a single linear layer. Common choices are ReLU (keep positives, zero negatives), sigmoid and tanh (squash into a range), and GELU or ELU (smoother variants). The choice affects how well gradients flow through deep networks.

Like a hinge in a ruler: one hinge lets it bend, many hinges let it trace any shape.

The parameters a mixture-of-experts model actually uses for one token, as opposed to all the parameters it stores.

Active parameters set the compute cost per token; total parameters set the memory needed. Mixtral 8x7B has about 47 billion total and 13 billion active; DeepSeek-V3 has 671 billion total and about 37 billion active.

Like the stations an order actually visits, versus every station in the kitchen.

A small function that converts one system's data shape into the shape your code uses.

Claude, OpenAI and Gemini put the same answer in differently shaped responses. One adapter per provider turns each into your own type, such as an `LLMReply` dataclass, so the rest of the app never touches provider-specific shapes. Adding a provider then means writing one adapter.

Like the unpacking station at the back door that pours every supplier's flour into the same bins.

A model running in a loop that decides its own next step, uses tools, checks results and keeps going until a task is done.

An agent is given a goal rather than step-by-step instructions. Each turn it chooses an action, often a tool call, observes the result and decides what to do next. Agents are powerful but harder to predict, test and secure than fixed workflows; Module 13 covers when to use them.

Like a personal assistant you hand a goal, who makes the calls and checks back, rather than one you direct step by step.

Wrapping a whole agent so another agent can call it like a function and get its answer back.

Inside, it is a full agent run with its own instructions and tools; the calling agent only sees the final answer and stays in charge of the conversation. It suits expert sub-questions and splitting work into focused parts. In the OpenAI Agents SDK you create one with agent.as_tool().

Like the counter assistant shouting a question to the head baker and relaying the answer.

A JSON document an A2A agent publishes to describe who it is, what skills it offers and how to talk to it.

It lives at a well-known address on the agent’s domain and lists its name, description, version, interface URLs, capabilities such as streaming, accepted input and output types, security schemes and skills with examples. Clients read it before sending any request. It advertises what an agent does, never how.

Like a business card and catalogue rolled into one, printed for software to read.

A library that packages common agent patterns, such as loops, tools and teams, so you write less code.

Examples include smolagents, AutoGen, CrewAI and LangGraph. Frameworks speed up prototypes but can hide prompts, add token overhead and change quickly between versions. Keep your tools and data access in plain code so you can switch frameworks without a rewrite.

Like an event company: faster than organising everything yourself, but you eat from their menu.

The repeating cycle at the heart of every agent: decide what to do, do it with a tool, look at the result, and decide again.

Your code sends the conversation to the model; if the model asks for a tool, your code runs it and appends the result; then it calls the model again. The loop ends when the model stops asking for tools or when your step limit is reached. Every agent library is a version of this loop.

Like a head chef at the pass who calls an order, hears what the runner found, and calls the next order.

A library that packages the agent loop with building blocks like tools, handoffs, guardrails and tracing.

Instead of writing the call-model, run-tools, repeat loop yourself, an agent SDK gives you agents, a runner, several kinds of tools, ways for agents to work together, and records of each run. The OpenAI Agents SDK and the Claude Agent SDK are two examples with similar concepts and different emphases.

Like a fitted kitchen with stations and equipment, instead of building every cupboard yourself.

Retrieval-augmented generation where the model decides when and how to search, and can search several times before answering.

Instead of one fixed retrieve-then-answer step, retrieval becomes a tool inside the agent loop. The model rewrites queries, follows leads from one passage to the next, applies filters and judges when it has enough evidence. It costs more and runs slower than classic RAG, and its judgement about when to stop is its weakest point.

Like a researcher who reads, notices a gap and goes back to the shelves, instead of a librarian who makes one trip.

A software engineer who builds useful, reliable products on top of AI models that others have trained.

Most of the work is around the model: writing instructions (prompts), getting the right facts to it (retrieval), connecting it to tools, measuring quality with evals, and shipping it fast, safely and affordably. It differs from ML research, which makes better models, and data science, which learns from data. The title became common around 2023.

Like a chef who turns excellent ingredients grown by others into a meal people want, served reliably every night.

The written rules, records and sign-offs that keep an AI system accountable.

Governance usually has four parts: a description of the system (a model card), a record of what it did (audit logs), rules about data (retention and privacy) and approvals for changes. Every claim should be backed by evidence such as a setting, a log or a test. A policy that no code enforces is worse than none, because people believe it.

Like a restaurant’s food safety file: allergen sheets, fridge temperature logs and signed-off recipes, ready for the inspector.

When an AI system treats some groups of people unfairly, often by repeating patterns in its data.

Bias can enter through training data, through inputs like speech transcription that works better for some accents, through how a task is framed, or through how outputs are used. It is found by testing, for example by changing things that should not matter and checking that results stay the same, and by monitoring real outcomes over time.

Like a judge who has only ever met people from one town and keeps favouring them without realising.

A simple plain-text prompt layout with ### Instruction: and ### Response: headings.

Popularised by Stanford's 2023 Alpaca project, it marks the instruction, optional input and response with ordinary text headings. It is easy to read and common in fine-tuning datasets, but has no real system prompt or multi-turn structure.

Like a worksheet with a "Question" box and an "Answer" box.

Flagging inputs that look unlike the normal data a model has seen.

A common approach trains a model only on normal data, then flags inputs it handles badly, such as those with high reconstruction error. The threshold is chosen from normal data, for example the level only the oddest 5% exceed. It flags “unusual”, not “wrong”, so it works best as a prompt for human review.

Like a shop assistant who has seen thousands of normal banknotes and notices when one feels different.

A score for whether an answer actually addresses the question that was asked.

An answer can be true and faithful but evasive or rambling. RAGAS measures relevancy by having a judge guess which questions the answer would reply to, then comparing those guesses with the real question using embeddings. Low scores usually point to the prompt or answer format.

Like a waiter who brings a perfectly cooked dish, just not the one you ordered.

A published doorway that lets one program ask another program to do something, in an agreed format.

Model providers expose their models through web APIs: your code sends a request over the internet and gets a response back. Most web APIs use HTTP and JSON. Every call to Claude, GPT or Gemini from your code goes through an API.

Like a bakery counter: you do not walk into the kitchen; you order in a set way and get your order back.

A long secret string that identifies your account when your code calls an API.

The provider uses the key to know who is calling and whom to bill. Anyone with your key can spend your money, so keep it in a .env file or a secrets manager, never in code, screenshots or git. If a key leaks, revoke it at the provider and create a new one.

Like the key to the shop's back door: anyone holding it gets in, and whatever they take goes on your tab.

When reviewers see so many approval requests that they start approving without really checking.

Approval fatigue turns human-in-the-loop review into a rubber stamp. It is avoided by tuning triggers so only genuinely risky actions need review, showing reviewers full context and the reason for each flag, and giving them real choices beyond "approve".

Like a manager asked to sign two hundred forms a day who stops reading them by lunchtime.

Finding vectors that are very probably the closest to a query, much faster than checking every vector.

Exact search compares a query with every stored vector, which becomes too slow at millions of vectors. ANN indexes such as IVF and HNSW check only a small, well-chosen fraction and accept occasionally missing a true neighbour. How often they miss is measured as recall, and settings like nprobe or ef_search trade recall for speed.

Like finding a book by going straight to the right section of the library instead of reading every spine, accepting that a misfiled book might be missed.

An output produced by an A2A task, made of parts such as text, structured data or files.

Artifacts let a remote agent return results as named fields, like a booking reference and a fee, rather than chatty sentences. Treat them as untrusted input: validate the values and never let their text instruct your agent to take risky actions.

Like the invoice and delivery note that come with a finished order.

Getting computers to do tasks that would seem to need intelligence if a person did them.

AI is the broadest umbrella term. It includes hand-written rules and search (like early chess programs) as well as systems that learn from data. Machine learning, deep learning and generative AI are all parts of AI, but AI is not only those.

Like the word "cooking": it covers everything from boiling an egg by a fixed rule to a chef improvising a new dish.

A way of writing Python so that, while one task waits, other tasks can make progress.

`async def` defines a coroutine function; `await` marks a point where it can pause, for example while waiting for an API. With `asyncio.gather`, many waits overlap, so total time approaches the slowest call instead of the sum. It speeds up waiting, not heavy calculation.

Like a baker who loads ten ovens and handles whichever timer rings first, instead of standing at one oven.

A system design where the request that starts work returns immediately and separate workers do the work later.

The web server accepts a request, puts a job on a queue and returns a job ID; workers process jobs at a controlled pace and store results. It survives restarts and bursts far better than a synchronous design. It is a system design, not the same thing as Python's async and await keywords.

Like a restaurant token system: order at the counter, get a number, the kitchen cooks, your number is called.

Python’s built-in toolkit for running many waiting tasks at the same time on one thread.

Functions defined with async def can pause at each await, letting other tasks run while they wait for the network. That makes asyncio ideal for LLM calls, which are mostly waiting. It does not use more CPU cores, and it only switches tasks at an await.

Like one cook watching eight pots, moving to the next whenever one is simmering.

A stored, categorised collection of attacks you can replay against any version of your system.

Each attack is kept as data: the input turns, a category, a severity and what must not happen. Seed attacks are multiplied into variants across languages, encodings, wrappers and turn counts. The library grows from red-team sessions and suspicious production conversations, and tools such as promptfoo, garak and PyRIT ship with large ready-made collections.

Like a locksmith’s collection of every lock-picking trick they have ever seen, used to test each new lock.

Of all attack attempts against a system, the share that made it do something it should not.

It is measured on outcomes, such as money moved, data leaked or emails sent, rather than on whether the model refused. Because models are not deterministic, each attack is run several times. Tracking it before and after a change shows whether defences actually work.

Like counting how many of the hired intruder's attempts actually got into the stockroom.

The part of a transformer that lets each word look at the other words and decide which ones matter to it.

For every token, attention computes how relevant every other token is and mixes in information from the relevant ones. This is how a model links "he" back to "my son" in a long message. Attention is the core idea of the transformer and is explained step by step in Module 30.

Like a reader's highlighter, marking which earlier words to keep in mind while reading the next one.

Removing background noise such as traffic or machines from a recording while keeping the voice.

Classic denoising estimates the steady background noise and subtracts it; neural enhancement models learn to separate speech from noise. Denoising usually helps people listen, but it can add artefacts that make speech recognition worse, so measure its effect on transcripts before adding it to a pipeline.

Like turning off the mixers in the kitchen so you can hear the customer on the phone.

A list of numbers that captures what a sound clip is like, so similar sounds get similar lists.

An audio model such as wav2vec2 produces one vector per short slice of audio; averaging them gives one vector per clip. Comparing these vectors with cosine similarity finds duplicates, groups recordings, or matches speakers when using a speaker-specific model. CLAP goes further and places sounds and text descriptions in the same space.

Like a fingerprint for a sound: two recordings of the same oven timer leave nearly the same print.

A permanent, time-stamped, append-only record of what a system did and who approved it.

Audit logs record decisions: model and prompt version, tool calls that changed data or money, results and human approvals. Entries are added, never edited, and are kept for a defined period even when nothing went wrong. They should hold references rather than personal details, and a hash chain can make tampering detectable.

Like the signed fridge temperature log in a kitchen: written at the time, and nobody can quietly change last Tuesday’s entry.

A single model call given helpers such as retrieval, tools or memory: the basic building block of every workflow and agent.

The phrase comes from Anthropic’s essay Building effective agents. Workflow patterns are different ways of wiring augmented LLM calls together, with your code deciding the wiring. Each call can have its own prompt, tools and effort level.

Like a single kitchen station with its own tools: the stations are combined in different ways to run the kitchen.

Helpers in the transformers library that read a model's settings file and load the right tokenizer or model for you.

Given a checkpoint name, AutoTokenizer, AutoModelFor... and their siblings open config.json, read the model_type, and build the matching architecture class. That lets the same code load BERT, Gemma or Whisper by changing one string. The suffix of AutoModelFor... chooses which task head is attached.

Like a receptionist who reads your booking slip and sends you to the right room without you knowing the building.

Automatically building worked reasoning examples by having the model solve representative questions.

Questions are grouped by similarity, one is picked from each group, and the model solves it step by step. The good solutions become few-shot chain-of-thought examples. Choosing from different groups keeps the examples diverse.

Like a teacher assembling a worked-examples sheet with one problem of each type, solved by the best student and checked.

A network trained to compress its input into a few numbers and rebuild it from them.

The encoder squeezes the input through a narrow bottleneck; the decoder reconstructs it. Because the target is the input itself, no labels are needed. The bottleneck forces the network to discover the most important features, which can be used for visualisation, compression or anomaly detection.

Like describing a cake over the phone in ten words so the baker at the other end can rebuild it.

An open-source multi-agent framework from Microsoft built around agents that talk to each other in conversations.

Its AgentChat layer provides assistant agents, teams such as round-robin and selector group chats, and termination conditions. It popularised conversational multi-agent systems. Its README now describes it as in maintenance mode and points new users to the Microsoft Agent Framework.

Like a meeting room where specialists take turns speaking until someone says the plan is agreed.

A framework feature that records your calculations and computes every gradient for you.

In PyTorch, tensors with requires_grad=True have their operations recorded. Calling .backward() on the loss replays the record in reverse, applying the chain rule, and stores each parameter’s gradient in .grad. It removes the need to derive gradient formulas by hand, which is where subtle bugs hide.

Like a sous-chef who writes down every step you take, then reads the notes backwards to tell you which step spoiled the dish.

A ladder describing how much control a system hands to the model, from a single call up to teams of agents.

Lower rungs (single calls, fixed chains, routers) keep the path in your code. Higher rungs (tool-using agents, long-running agents, multi-agent systems) let the model choose more of its own steps. Each rung up adds flexibility but also cost, latency and testing difficulty, so most products live on the lower rungs.

Like handing a new employee more responsibility one level at a time, from following a checklist to running a project.

A platform automatically adding or removing copies of your app to match traffic.

Autoscaling is controlled by concurrency (requests per copy), minimum instances (zero means scale to zero when idle) and maximum instances (a ceiling that caps cost). Apps that mostly wait on an LLM can handle many requests per copy. Without a maximum, a spike or attack can scale costs without limit.

Like a food court opening more counters when the lunch queue grows and closing them in the quiet afternoon.

An extra training objective added to help the main task learn, even if its output is not the main goal.

An auxiliary head predicts something related, such as the type of a receipt field, from the same shared features. Its loss, usually given a small weight, pushes the shared layers to learn useful structure. Whether it helps must be checked with ablations at several weights, including zero.

Like a trainee baker also being asked to name each pastry, which sharpens how they look at dough.

Grouping open-coding notes into a small set of named failure categories and counting them.

Axial coding is the second pass of error analysis. You sort the notes into buckets, create a new bucket when nothing fits, merge buckets that are the same thing, and name each one specifically enough to suggest a fix. Counting the buckets turns a pile of notes into priorities.

Like sorting a heap of post into pigeonholes, then counting how many letters landed in each.

Translating a translation back into the original language to check that the meaning survived.

A separate call translates the result back, and you compare it with the source. It catches missing sentences, wrong numbers and dropped warnings quickly. It cannot judge whether the translation sounds natural, so important text still needs a fluent reader.

Like repeating an order back to the customer to make sure you heard it right.

Pushing back on new work when a system is already too far behind, instead of accepting work it cannot finish in time.

If the queue would make new customers wait longer than they will stay, the API should refuse politely, for example with HTTP 503 and a Retry-After header, or send a holding message. Choose the limit from the acceptable wait multiplied by how fast workers finish jobs.

Like a bakery putting up a "sold out for today, back tomorrow" sign rather than taking orders it cannot bake.

The method that works out every parameter’s gradient by applying the chain rule backwards from the loss.

Backpropagation starts at the loss and passes blame back through each layer, multiplying local slopes along the way, until every weight knows its share. It costs roughly one extra forward pass, which is why training networks with billions of parameters is possible at all. In PyTorch it runs when you call loss.backward().

Like a customer complaint travelling back from the counter to the oven to the proofing room, each step passing on its share of the blame.

Training a recurrent network by unrolling it across time steps and backpropagating through every step.

The same weights are used at every step, so their gradient adds up contributions from all steps. The gradient is multiplied at each step on the way back, which is why it vanishes or explodes over long sequences. Gradient clipping and gated cells such as LSTMs make it workable.

Like tracing a mistake back through every page of a long logbook, one page at a time.

Running a trading strategy on past data to see how it would have performed.

Backtesting is the only way to learn about a strategy before risking money, and the easiest place to fool yourself. Honest backtests avoid look-ahead and survivorship bias, subtract realistic costs, report sample sizes, compare against a simple baseline, and keep some data aside for a final, untouched check.

Like rehearsing a play with last year’s audience reactions: useful practice, but no guarantee of tonight’s applause.

Letting a caller interrupt a voice agent, which must stop talking straight away and listen.

When the VAD hears the caller start speaking during playback, the agent stops its audio, cancels the rest of its reply, and records in its history only the part the caller actually heard. False barge-in from echo or "mm-hmm" sounds is avoided with echo cancellation and a minimum speech length.

Like a shop assistant who stops mid-sentence the moment a customer says "Actually, wait".

A model trained only to predict the next token on large amounts of text, before any instruction tuning.

A base model is a powerful autocomplete: give it a question and it may continue with more questions. It does not reliably follow instructions or stop when done. Chat and instruct versions are built on top of base models.

Like a very well-read person who finishes your sentences but has never been taught to answer questions.

A way of writing any file, such as an image, as plain text characters so it can travel inside JSON.

APIs that accept images inside a request often expect the file's bytes encoded as base64 text. The encoded text is about a third larger than the original file, so shrinking images before encoding saves upload time. Decoding turns the text back into the exact original bytes.

Like reading a photo out over the phone as a long list of letters and numbers that the other person can turn back into the same photo.

A provider service that runs many requests in the background at a discount.

You submit thousands of requests at once, each with your own id, and collect the results later, usually within an hour and at most within 24 hours. Anthropic and OpenAI charge about half the normal price for batch work. Results can arrive in any order, so match them by id.

Like a wholesale order the bakery fills overnight when the ovens are free, at a lower price.

A layer that re-standardises its inputs within each batch so every layer sees values on a stable scale.

BatchNorm subtracts the batch mean and divides by the batch standard deviation for each feature, then applies a learned scale and shift. It makes deep networks train faster and tolerate higher learning rates. At evaluation time it uses running averages collected during training, so model.eval() matters; it behaves poorly with very small batches, where LayerNorm (used in transformers) is preferred.

Like re-calibrating the scales at every station on the production line, not just at the start.

Sending or processing many items together in one go instead of one at a time.

Embedding APIs accept a list of texts per request, and local models run faster on groups of texts. Sending batches of around 100 texts turns 41,000 requests into about 410, which is far faster and avoids rate limits. Results come back in the same order as the inputs.

Like taking a stack of pages to the print shop in one trip instead of walking there once per page.

A fixed set of test questions with known answers, used to compare models.

Benchmarks cover school-style exams, maths, coding and more, and leaderboards rank models by them. They are useful for shortlisting but test other people's tasks, can leak into training data, and ignore price and speed. Your own eval on your real examples decides.

Like a standardised exam: a useful signal, but not the same as watching a candidate do your actual job.

A 2018 encoder-only transformer from Google that reads text in both directions and is used for understanding tasks.

BERT was pretrained with masked language modelling and is typically fine-tuned for classification, tagging or search. Many embedding models and rerankers are BERT-style encoders. Classic BERT models accept at most 512 tokens.

Like a sharp-eyed proofreader: brilliant at understanding a page, not hired to write one.

A model that turns the question and each document into embeddings separately, so documents can be embedded in advance.

Standard embedding search uses a bi-encoder: chunks are embedded once at indexing time and only the question is embedded at query time, which makes search fast. The price is that the model never reads the question and a chunk together, so it can miss subtle mismatches that a cross-encoder catches.

Like matching people by their separately written profiles rather than meeting them together.

An evaluation that asks one yes-or-no question about one output, such as "did the reply include the allergy warning?".

A binary eval checks a single failure mode against a written definition of pass and fail, with examples. Run across many outputs, it gives a pass rate. Binary evals are easier to agree on, harder to game and more actionable than 1-to-5 scores.

Like a food safety inspector's checklist: fridge below 5°C, yes or no.

How much damage a failure can cause: how many users, how much money or data, for how long.

Production risk work is mostly about shrinking the blast radius in advance, since failures cannot be fully prevented. Tool limits, kill switches, rate limits, staged rollouts and human approvals each reduce it. Asking "what is the worst this could do tonight?" for every component is a useful habit.

Like a fuse box: a fault trips one circuit and the rest of the house keeps its lights.

A classic keyword-search scoring method that ranks documents by how well their words match a query.

BM25 gives more weight to rare words than common ones and avoids over-rewarding long documents. It needs no model and works well for small collections, but it cannot match meaning: "money back" does not find "refund". Embeddings and hybrid search address that.

Like a librarian who finds books by matching the exact words on your request slip.

A rectangle around an object in an image, described by its left, top, right and bottom pixel positions.

Object detectors describe where each object is with a bounding box, often given as xmin, ymin, xmax and ymax in pixels. Boxes are cheap to compute and draw, but they include some background around irregular shapes. When you need the exact outline, use segmentation instead.

Like drawing a rectangle around a doughnut on a printed photo with a marker.

A hosted platform built mainly for testing and scoring LLM app outputs.

Braintrust is designed evaluation-first: you run your app over datasets, grade outputs with scorers and compare experiments side by side. It also logs and traces production traffic and offers a prompt playground. The platform is not open source, though its `autoevals` scoring library is, and enterprise customers can keep data in their own cloud.

Like a test kitchen where every new recipe is baked, tasted and scored against the old one before it reaches the shops.

A separate line of commits where you can try changes without affecting the main version.

The main branch, usually `main`, should always work. You create a branch for each experiment or feature with `git switch -c name`, commit on it freely, and merge it back when it is ready. Branches also let several people work at once without overwriting each other.

Like a test kitchen where a new filling is tried out before it goes on the production line.

NumPy's rule for combining arrays of different shapes by stretching the smaller one.

Shapes are compared from the right: each pair of dimensions must be equal, or one must be 1. A `(3, 4)` grid times a `(4,)` row works; times a `(3,)` row fails until you reshape it to `(3, 1)`. Shape errors are the most common bug in numerical code.

Like applying one row of daily prices to every store's row of sales.

An agent that completes tasks on websites by controlling a web browser.

It may see the page as screenshots, as a text view built from the accessibility tree, or both. It is useful when a site has no API, but it inherits every risk of the web, from pop-ups to prompt injection. Scripted steps, allowlists and human checks keep it safe.

Like an intern who can use any website you point them at, quickly and without getting bored, but a little gullible.

The most common way to build a tokenizer: start with single characters and repeatedly merge the pairs that appear most often.

BPE counts which neighbouring pieces occur most frequently in a large body of text, merges the top pair into a new token, and repeats until the vocabulary is the desired size. Frequent words end up as single tokens and rare words are built from smaller pieces. Because it starts from raw bytes, any text, in any language, can be represented.

Like a shop that notices "almond flour 500 g" is ordered constantly and gives it one short code, while rare items are written out piece by piece.

A way of building a tokenizer by repeatedly gluing the most common pair of neighbouring pieces into a new token.

Byte-level BPE starts from the 256 possible bytes, so it can represent any text, then learns an ordered list of merges from training text. Frequent strings become single tokens while rare text stays split into many pieces. Encoding replays the merges in the order they were learned.

Like kitchen staff inventing shorthand for the orders they write most ("ELC" for eggless chocolate cake) and spelling out everything else.

A marker in an Anthropic request that says "save everything up to here" in the prompt cache.

You add `"cache_control": {"type": "ephemeral"}` to a content block, and the provider caches the whole prompt up to and including that block. The cache key is the exact text up to the breakpoint, so it belongs at the end of the part shared between requests, never after a unique question. A request can have up to four breakpoints, and a top-level `cache_control` places one automatically on the last block for growing conversations.

Like a bookmark in a recipe binder that says "everything before this page is the same for every order".

A request that found its prompt prefix already saved and reused it instead of computing it again.

A cache hit happens when a request starts with exactly the same text as a recent request that saved a cache entry. On Anthropic it shows up as `cache_read_input_tokens` in the usage; on OpenAI as `cached_tokens`. Hits are billed at a small fraction of the normal input price. The opposite, a miss, means the prefix was processed in full and possibly written to the cache.

Like finding the dough you prepared earlier still in the fridge, instead of making it again from flour.

The share of your input tokens that were served from the prompt cache instead of being processed in full.

You compute it from the usage fields over many requests: tokens read from the cache divided by the sum of tokens read, tokens written and uncached input tokens. A high rate (above 80 or 90 percent for a busy app with a long shared prefix) means the layout works. A sudden drop after a deployment usually means a silent invalidator, such as a timestamp or a reordered tool list, has crept into the prefix.

Like the share of orders a bakery fills from stock already on the shelf rather than baking from scratch.

How well a stated confidence matches how often the answer is actually right.

A well-calibrated system that says 90% is right about nine times in ten. Numbers a language model writes about its own confidence are often poorly calibrated. Measurable signals, such as disagreement between runs or unusual inputs, are better triggers for human review.

Like a weather forecaster whose "70% chance of rain" days really are rainy about 70% of the time.

Releasing a new version to a small share of users first, and widening it only if the numbers stay healthy.

A canary sends, say, 5% of traffic to the new version while you watch error rates, feedback and cost, then moves to 25% and 100%, or rolls back. Canaries catch problems in common traffic. They are too small to catch rare but dangerous cases, which must be caught earlier by evals.

Like a bakery selling a new recipe in one store for a week before rolling it out to all six.

How an MCP client and server agree on a protocol version and on which features each supports.

In the 2025 revisions of MCP this is a handshake: the client sends initialize, the server replies with its version and capabilities (tools, resources, prompts), and the client confirms with an initialized notification. The stateless 2026-07-28 revision instead attaches the version and client capabilities to every request, with an optional server/discover request to learn what a server offers. Either way, neither side uses a feature the other did not declare.

Like two people agreeing which language to speak and what each can help with before getting down to business.

When training a model hard on a narrow task makes it lose general skills it had before.

Fine-tuning moves the weights towards the new data, which can wear away earlier abilities such as ordinary conversation or other languages. Small learning rates, few epochs, mixing in general data and training only small adapters all reduce the risk.

Like a chef who spends a year making only croissants and forgets how to make a good dal.

A training game where the model predicts each next token using only the tokens before it.

Thanks to the causal mask, every position in a text is a graded prediction in a single pass, which makes it data-efficient. It is how GPT-style models are pretrained, and it directly practises the skill of writing.

Like learning to tell a story by guessing, again and again, how the next sentence begins.

The average point of a group of vectors: its centre.

You get a centroid by averaging a group of vectors position by position. A customer’s centroid represents their overall taste, and a label’s centroid represents a typical example of that label. The average of unit vectors is shorter than 1, so normalise it before comparing, and remember that a group with several separate kinds of item has a centroid that matches none of them well.

Like the "typical letter" you would picture for each pigeonhole in a post office.

When A affects B and B affects C, the effect of A on C is the two slopes multiplied together.

The chain rule from calculus is what lets a network work out how a weight deep inside it affected the final loss. Backpropagation applies it layer by layer from the output back to the input. Multiplying many slopes together is also why gradients can vanish or explode in deep networks.

Like a complaint passed back through a kitchen, each station passing on its share of the blame.

Asking a model to write out its reasoning steps before giving its final answer.

A model can only "think" through the text it writes, so written steps act as its scratch paper and improve multi-step problems. Put the reasoning and the answer in separate tags so code reads only the answer. Reasoning models now do this internally.

Like doing long division on paper instead of in your head.

Tool calls made one after another because each needs the previous result.

If the model must find an order’s store before checking that store’s stock, it requests the first tool, reads the result, then requests the second in its next turn. The same tool loop handles chained and parallel calls.

Like finding out which branch has your parcel before phoning that branch.

A voice agent built from three stages: speech-to-text, a language model, and text-to-speech.

Because text sits in the middle, a chained agent can use any language model and tools, any voice, and keeps a transcript of every turn. Run one stage after another it is slow; streamed and pipelined, so the first sentence is spoken while the rest is generated, it typically replies in about one to one and a half seconds. Mistakes in transcription can compound down the chain.

Like a bakery production line where baking, packing and labelling all run at once on different trays.

The share of characters an OCR system got wrong, counting insertions, deletions and substitutions.

CER is the edit distance between predicted and true text divided by the length of the true text. A CER of 2% means about one character wrong in fifty. It is usually reported alongside exact-match accuracy, since one wrong digit can make a whole invoice total useless.

Like counting the typos a proofreader would have to fix per hundred letters.

A detailed description of a role-play persona: scenario, personality, goals, hidden state and limits.

A thin persona like "be angry" collapses or never changes. A character sheet gives the model a situation, rules for how its state changes in response to the user, an end condition and boundaries, which makes role-play realistic and measurable.

Like the brief an actor gets before an improvised training scene.

The repeating cycle at the heart of every chatbot: read a message, send the history, show the reply, save it, repeat.

Each turn appends the user message, sends the system prompt plus history to the model (often streaming the reply), appends the assistant reply, records token usage and trims if needed. Commands such as /reset and error handling wrap around that core.

Like the rhythm at a busy counter: listen, check the folder, answer, note it down, next.

One entry in a conversation sent to a chat API, made of a role and its content.

In Python a message is usually a small dictionary such as {"role": "user", "content": "Do you deliver to Aundh?"}. Content can be a plain string or a list of pieces such as text and images. The list of messages you send is the whole truth for that request.

Like one line in a transcript, with the speaker named at the start.

An LLM that has been trained further to answer questions and follow instructions helpfully, instead of just continuing text.

After pretraining, model makers train on many example conversations and on human ratings of answers, so the model learns to reply like a helpful assistant and to decline harmful requests. GPT, Gemini and Claude as you use them through their APIs are chat models. Underneath, they still predict the next token.

Like the same well-read chef after a month of training at the counter: same knowledge, now trained to serve customers.

Labels on each message in a conversation saying who is speaking: your instructions, the person, or the model.

The three everyday roles are system (your standing instructions), user (the person chatting) and assistant (the model's own replies); Gemini calls the assistant "model". A tool role for function results appears later. Models are trained to treat each role differently, giving the system role the most weight.

Like the speaker names in a play script, set apart from the lines so no actor can claim to be the director.

The format a provider or model uses to turn a list of messages into one sequence of tokens with role markers.

Behind the API, messages are wrapped in special tokens that mark where each turn starts and ends and who is speaking; ChatML is one public example. Closed providers apply their own hidden templates for you. When you run open models yourself, using the right template matters.

Like the house style a typist uses to turn a recorded conversation into a formatted transcript.

A chat template that wraps each turn in <|im_start|>role and <|im_end|> markers.

ChatML came from OpenAI and is used by several open model families, including Qwen. The prompt ends with an open assistant turn as the cue to answer, and the model ends its turn with <|im_end|>. It supports system prompts and multi-turn chat cleanly.

Like name cards placed in front of each speaker at a panel discussion.

A saved copy of a trained model's weights and settings that you can load later.

In Hugging Face terms a checkpoint is a Hub repo or local folder containing weights, config.json and tokenizer files, named like distilbert/distilbert-base-uncased-finetuned-sst-2-english. Different checkpoints of the same architecture can be trained for different tasks. During training, checkpoints are also the periodic saves you can resume from.

Like a saved game: everything the model learned up to that point, ready to load and continue.

The component that saves a snapshot of a graph's state after every step so a run can be inspected or resumed later.

You pass one to compile(checkpointer=...). Each snapshot, called a checkpoint, records the state and which node runs next, filed under a thread ID. InMemorySaver keeps them in the process and is lost on restart; database-backed savers such as MongoDBSaver survive restarts and can be shared by several servers. Checkpointers also make interrupts and time travel possible.

Like the auto-save in a video game, written to a memory card that any console can read.

An open-source vector database that can run inside a Python script or as a small server.

Chroma stores records (ID, embedding, document and metadata) in collections, embeds text for you with a default local model, and supports metadata and text filters plus add, upsert, get, update and delete. It is the quickest way to start, and it scales comfortably to millions of vectors on one machine.

Like a well-organised filing clerk who sits at your own desk and works for free.

How much text neighbouring chunks may share, so a sentence cut at a boundary still appears whole in one of them.

When a splitter must cut inside a long passage, overlap repeats a little of the end of one chunk at the start of the next. It is insurance against slicing a rule in half. Recursive splitters build overlap from whole pieces, so when every piece is longer than the overlap, chunks may not overlap at all.

Like overlapping roof tiles: each covers the edge of the next so no gap lets the rain through.

Cutting long documents into smaller passages so the relevant piece can be found and sent to a model.

Chunks are often a few hundred words, sometimes overlapping so ideas are not cut in half. Chunk size is a real trade-off: too small loses context, too large wastes tokens and blurs search results. A badly placed cut can hide exactly the fact you need.

Like slicing a cake so you can serve one piece without carrying the whole cake to the table.

Automatically checking every change and releasing passing changes through one repeatable process.

Continuous integration runs linting, tests and, for AI apps, an eval gate on every proposed change. Continuous delivery or deployment builds and releases changes that pass, without anyone typing commands on a laptop. Checks are ordered from cheap to expensive, and the pipeline becomes the only path to production.

Like a bakery quality line where every tray passes the same checks in the same order, and nothing goes round the side door.

A switch in your code that stops calling a failing service for a while after repeated errors.

A circuit breaker has three states: closed (normal), open (skip the service entirely for a cooldown after, say, five failures in a row) and half-open (let one trial request through to test recovery). It spares users the timeout penalty during outages and gives the struggling service room to recover.

Like the trip switch in your fuse box: it cuts the line when something shorts, and you flip it back later to test.

A pointer from a claim in an answer to the exact source passage it came from.

In RAG systems, retrieved passages are numbered and the model marks each claim with a number such as [2]. Code then turns the numbers into links, for example to a document page or a video timestamp, and drops any number that does not match a real source. Citations let people check answers quickly.

Like a footnote in a textbook that tells you which page of which book a fact came from.

The audio version of CLIP: it places sounds and text descriptions in the same embedding space.

CLAP pairs an audio encoder with a text encoder trained on sound clips and their descriptions. You can classify a sound against labels such as "an oven timer beeping" without training, or search a sound library with words. Check the expected sampling rate, often 48 kHz, on the model card.

Like a sound engineer who can tell you which of your written descriptions matches the noise coming from the kitchen.

A blueprint that describes what a kind of thing has and what it can do.

A class bundles data (attributes) with the functions that work on it (methods). You create objects from it by calling it, like `Order("CR-1042", "Baner")`. Class names use CapWords.

Like a cake mould: it is not a cake, but every cake made from it has the same shape.

Assigning each item to one of a fixed set of labels.

Tagging reviews by category or routing tickets by team are classification tasks. LLMs can classify from instructions and examples, and structured outputs keep their answers to the allowed labels. You measure classifiers on a hand-labelled answer key.

Like sorting the morning post into labelled pigeonholes.

A trick that makes a diffusion model follow its text prompt more strongly.

At each step the model predicts the noise with and without the prompt and pushes further in the direction the prompt suggests, scaled by a guidance weight. Higher weights follow the prompt more literally but can look overcooked.

Like turning up the flavour you were asked for, compared with the plain version.

Anthropic's SDK that gives you the agent harness behind Claude Code to build your own agents.

It starts from an agent that can already work with files, run commands and fetch web pages, which you customise with a system prompt, allowed tools, your own tools and MCP servers, subagents, hooks and permission modes. It suits coding, research and document-heavy automation.

Like hiring an experienced assistant who already knows the office tools, then giving them your house rules.

A program you run from the terminal, controlled with typed options.

A CLI like `uv run triage.py email.txt --provider claude --dry-run` lets teammates use a workflow without editing code. Python's built-in `argparse` turns options into variables and writes the `--help` text. Return exit code 0 for success and non-zero for failure.

Like a well-labelled control panel instead of rewiring the machine each time.

A model that places images and text in the same embedding space, so you can match any picture with any caption.

CLIP has an image encoder and a text encoder trained together on about 400 million image-caption pairs, so a photo and its true caption land close together. That enables zero-shot image classification, multi-label tagging with thresholds and searching photos with sentences. It cannot see invisible properties, is easily swayed by text visible in the image, and reflects the biases of its internet training data.

Like a librarian who has looked at millions of captioned photos and can tell you which of your captions fits a new picture best.

A model whose internal parameters are kept private, so you can only use it through the provider's API.

GPT, Gemini and Claude are closed. You get frontier quality with no hardware to run, but your data is sent to the provider and you depend on their prices, terms and model retirements. The alternative is an open-weights model you can download and host.

Like a famous bakery's secret recipe: you can buy the cake, but not see or change the recipe.

An agent that takes actions by writing short programs that call its tools, instead of one tool request at a time.

Because its actions are real code, a code agent can loop, store results in variables and calculate exactly, often finishing in fewer steps. The cost is safety: model-written code is untrusted and must run in a restricted interpreter or, for real use, a sandbox.

Like a manager who hands over a written checklist instead of sending one text message per instruction.

Using a model to write, change or explain code from a description.

Models write code well because they were trained on a great deal of it, but they fill gaps with plausible guesses, such as columns that do not exist. Give them the real schema, house rules and versions, ask for assumptions and diffs, and check results with tests or known totals.

Like a very fast contractor on their first day: brilliant, but new to your codebase.

An eval checked by ordinary code, such as a string search or a lookup, with no model involved.

Code-based evals are fast, free and give the same answer every time. They suit checks like "is the output valid JSON?", "does the status match the tool result?" or "is the reply under 80 words?". Use them wherever possible and save LLM judges for real judgement calls.

Like checking a cake's weight on a scale instead of asking someone how heavy it feels.

An AI agent that reads a codebase, edits files and runs commands such as tests until a programming task is done.

A coding agent is an agent loop with tools for listing, reading and searching files, making precise edits and running commands, usually inside a sandbox. Good ones verify their work by running tests rather than trusting the model's claim, ask a human before risky actions and manage their context so they can work through long sessions. Claude Code is a well-known example.

Like a new developer who can only work in a locked practice room and must show passing tests before going home.

The difficulty of recommending new items or serving new users when there is no history about them yet.

Behaviour-based recommenders cannot suggest an item nobody has bought yet, or personalise for a customer who has never ordered. Content-based methods with embeddings help, because a new item’s description already gives it a vector. Many systems start content-based and add behaviour as history builds up.

Like a new dish on the menu that no regular has tried yet, so nobody can say "people who liked this also liked...".

A named group of records in a vector database that share one embedding model and one distance setting.

Chroma and Milvus organise data into collections, roughly like tables in an ordinary database. Each collection has its own index. Because vectors from different models cannot be compared, one collection should only ever hold vectors from one embedding model.

Like one drawer in a filing cabinet, holding folders that are all filed the same way.

A list of the only programs an agent is permitted to run; everything else is refused.

Before running a command, the agent's tool splits it with shlex.split and checks the first word against the allowlist, after rejecting shell operators such as ;, &&, | and > that could chain extra commands. An allowlist keeps the agent on an expected path and makes surprises visible, but it is not a security boundary on its own; the sandbox is.

Like a guest list at the door: if your name is not on it, you do not get in, however politely you ask.

A saved snapshot of your project in git, with a message saying what changed and why.

Each commit records the state of the tracked files, the author, the time and a unique ID called a hash. You choose what goes in with `git add` and save it with `git commit`. Commit whenever something works, before you start improving it.

Like a save point in a video game, made just before the boss fight.

One building block on a Langflow canvas, with inputs on one side and outputs on the other.

Components include Chat Input, Prompt, model components, File, Split Text, vector stores, Message History and Agent. You connect an output to a compatible input to build a flow, and each component's settings live in its panel. Component names change between Langflow versions, so search the sidebar when a name in a tutorial does not match.

Like a single Lego brick with studs that only fit the right kind of brick.

Building an object out of other objects it holds and uses.

An `Order` has `OrderLine`s; a `Conversation` has an LLM client. Composition models "has a" relationships and makes parts swappable, such as a fake model in tests and a real one in production. It is usually safer than deep inheritance.

Like a Diwali hamper: it contains sweets and candles, but it is not a kind of sweet.

A one-line way to build a new list, dictionary or set from another collection.

`[p * 2 for p in prices if p < 100]` keeps the prices under 100 and doubles each one. The same pattern builds dictionaries and sets. Keep comprehensions simple; if one needs explaining, write a loop.

Like a conveyor belt with a sieve and a stamp: drop the ones you do not want, ice the rest.

Letting a model operate a computer by looking at screenshots and choosing clicks, typing and scrolling.

It works on any software with a screen, including old websites and desktop programs with no API. It is slow and expensive because each step needs a fresh screenshot, and risky because anything on screen can contain instructions. Run it in a sandbox with domain allowlists and human approval for consequential actions.

Like helping someone with a website over a video call: look, tell them where to click, look again.

The part of AI that turns images and video into useful information, like what is in a photo and where.

Computer vision covers tasks such as image classification, object detection, segmentation, reading text from images and understanding video. Modern vision models learn visual patterns from millions of labelled images rather than following hand-written rules. They work best on images similar to the ones they were trained on.

Like teaching a new staff member to recognise every product in the display case by showing them thousands of photos.

When the right answer for the same inputs changes, so a model’s learned rules stop being true.

With concept drift the inputs can look exactly as before, but the relationship between inputs and outcomes has moved, for example because a competitor opened next door. It is usually invisible in the inputs, so you catch it by comparing predictions with real outcomes. Fixes include retraining on recent data and adding new inputs that capture the change.

Like a map that is still drawn perfectly, but the city has rerouted the roads.

Having many tasks in progress at the same time, taking turns.

Concurrency overlaps waiting: several requests are in flight while one thread switches between them. It differs from parallelism, where tasks literally run at the same instant on different processor cores. Async Python is concurrency on one core.

Like one waiter serving ten tables by moving between them, rather than ten waiters.

An arrow with a decision in it: a small function looks at the state and picks which step runs next.

You add one with add_conditional_edges(source, router, mapping). After the source node runs, LangGraph calls the router with the state; the router returns a label, and the mapping turns that label into a node name. Routers should only choose, never change state, and should always validate labels so an unexpected value falls back to a safe branch.

Like the quality-check station deciding whether a cake goes back to decorating or on to packing.

A prompt whose sections are included or left out depending on facts your code already knows.

Instead of one giant prompt with "if this, do that" written in English, code chooses which sections apply (wholesale client, festival order, allergy question) and renders only those. The model sees shorter, more relevant instructions, which is cheaper and usually more accurate.

Like a baker adding sesame or raisins to the same base dough depending on which loaf is on the order.

A table counting every combination of what the model predicted and what actually happened.

For a yes/no problem it has four cells: true negatives, false positives (false alarms), false negatives (misses) and true positives. For many classes it shows which classes get mistaken for which. It reveals mistakes that a single accuracy number hides, especially on imbalanced data.

Like a tally sheet of every right call, false alarm and miss a referee made in a season.

Steering a model while it generates so it can only produce tokens that keep the output valid.

At each step, tokens that would break the required format (for example a letter where a digit must go) are blocked before one is chosen. This is how strict structured outputs guarantee a schema. It controls shape, not truth: among the valid options, the model can still choose the wrong one.

Like a keypad that only lights up the buttons you are allowed to press next.

The method that sets up a new object when it is created.

In Python it is called `__init__`, and Python runs it automatically when you create an object. It usually stores the starting data on `self`. Dataclasses write it for you.

Like filling in the order slip the moment a new cake goes into the oven.

One isolated, running copy of a Docker image.

A container has its own files, processes and network identity, separate from your laptop and from other containers. Inside a container, localhost means the container itself. Anything written inside it disappears when it is removed, unless stored in a volume.

Like one loaf baked from the frozen dough: identical to the others at the start, then its own thing.

Deciding exactly what information goes into a model's context window on each call, in what order and form.

In real applications most of the context is assembled by code: instructions, pinned facts, retrieved documents, summaries and recent messages. Context engineering chooses the relevant pieces, orders them so caching works and vital facts stay prominent, and keeps the total small. It is now seen as the core design work of LLM apps.

Like laying out exactly the right papers on a colleague's desk before each call, and clearing away the rest.

Giving each agent or subagent its own separate context, so raw material read by one does not clutter another.

A specialist or subagent can read tens of thousands of tokens and pass back only a short summary, keeping the main agent’s context small and focused. This is the biggest reason to split work across agents. The trade-off is that detail is lost in the summary.

Like separate desks for each reporter, so the editor’s desk holds only finished memos.

Keeping what an agent sends to the model small and relevant, so long sessions stay affordable and accurate.

Techniques include clipping tool output, reading files in windows, summarising older messages when the conversation passes a size threshold (compaction) and keeping a notes file the agent updates itself. When removing old messages, a tool request and its result must stay together. Without it, every step pays for the whole history and important facts get buried.

Like clearing a workbench as you go and keeping a notebook of what you have done, instead of piling every offcut on the bench.

Something used with `with` that sets up a resource and always cleans it up afterwards.

`with open(path) as f:` opens a file and guarantees it is closed when the block ends, even if an error happens. HTTP clients and database connections work the same way.

Like a librarian who makes sure every borrowed book goes back on the shelf.

A score for whether the useful retrieved chunks are ranked near the top rather than buried under irrelevant ones.

Each retrieved position is judged relevant or not, and relevant chunks that appear early score higher. Low context precision means the right facts arrived but surrounded by noise, which distracts the model. Reranking and better chunk sizes are the usual fixes.

Like a delivery where the ingredient you need is at the bottom of five boxes of things you did not order.

A score for how much of the information needed for the correct answer was actually retrieved.

The reference answer is broken into claims, and each is checked against the retrieved chunks. Low recall means facts never reached the model, so no prompt change can fix the answer. Look at loading, chunking, search method, k and query rewriting.

Like checking whether every ingredient on the recipe actually arrived in the kitchen.

The gradual drop in answer quality as a context grows longer and noisier, even within the limit.

More tokens give the model more to sift through and more chances to be distracted. A focused context of a few thousand tokens often beats a sprawling one that technically fits. Bigger context windows raise the ceiling but do not remove this effect.

Like a desk so covered in papers that the important one is there but impossible to find.

The maximum amount of text, counted in tokens, that a model can consider at once.

The context window includes everything: instructions, retrieved documents, the conversation so far and the answer being written. Modern models range from tens of thousands to over a million tokens. When a conversation outgrows it, older parts must be dropped or summarised.

Like the size of a baker's worktable: everything must fit on it, and whatever falls off is forgotten.

A short line added to the top of a chunk saying where it came from, such as the document and chapter.

A chunk cut from the middle of a chapter often does not say what it is about. Prefixing it with something like "Store Operations Handbook > Chapter 9: Closing the store" before embedding helps search match it to questions. In the lesson, one header line moved the right chunk from second place to first.

Like writing the recipe name at the top of every page you photocopy from a cookbook.

A way of serving a model where finished conversations leave the batch and waiting ones join after every decode step.

With static batching, a group of requests runs until the longest one finishes, so slots sit empty and new requests queue. Continuous batching refills the batch at every step, keeping the GPU busy and letting new requests start almost at once. Serving engines such as vLLM are built around it, and it is a major reason hosted models can offer low prices per token.

Like a restaurant that seats the next customer the moment a table frees up, instead of waiting for the whole room to finish dinner.

A server that automatically runs your tests every time someone proposes a code change.

CI services such as GitHub Actions run your test suite on each pull request and block changes that break it. For LLM apps, fast tests run on every change, while slower eval tests with a real model run when prompts or models change. API keys live in the service's encrypted secrets.

Like a quality check at the kitchen door that every tray must pass before it reaches the shop.

A way of training a model by pulling matching pairs of examples together and pushing unrelated ones apart.

Embedding models are usually trained contrastively: they see pairs that belong together, such as a question and its answer, alongside unrelated texts. If the matching pair is not clearly closer than the unrelated text, the model’s settings are nudged. Repeated hundreds of millions of times, this produces a space where paraphrases sit side by side.

Like teaching a child to sort socks by repeatedly showing them a matching pair next to an odd sock and saying "these two go together, that one does not".

A fixed list of allowed words, such as the only relationship types an extractor may use.

Without one, an LLM invents many names for the same idea (ALLERGIC_TO, HAS_ALLERGY, CANNOT_EAT), splitting the graph. A typed schema with a fixed list of options, plus a database schema that rejects anything else, keeps the vocabulary controlled.

Like a librarian's list of subject headings: every cookbook goes under "Cooking", never "Cookery" or "Recipes".

The list of earlier messages in a chat that your app resends with each new request.

Because the model is stateless, the history is how a chatbot remembers what was said. It grows every turn and is billed as input every time it is sent, so long chats get expensive and eventually need trimming or summarising.

Like the customer's folder of past messages that you hand the clerk before each reply.

A short, model-written summary of the older part of a chat, sent in place of the full messages.

When history is trimmed, a separate model call folds the dropped messages into a running summary that rides along with later requests. It keeps the gist of long chats cheaply, but it is generated text and can lose or distort details, so safety-critical facts should be pinned by code instead.

Like a manager's one-page note on top of a thick file: very useful, and only as good as the person who wrote it.

Sliding a small grid of learned weights across an image and computing a weighted sum at each position.

A convolution filter (or kernel), often 3×3, detects one pattern, such as a vertical edge, everywhere in the image. Stride is how far it moves each step; padding adds a border so edges are covered and the output size is kept. A layer learns many filters at once, each producing its own output channel.

Like a stencil you slide over a tray, scoring each spot by how well it matches the stencil’s shape.

A neural network built from small filters that slide across an image, the standard architecture for vision.

CNNs stack convolution layers, activations and pooling, so early layers detect edges and textures and later layers detect parts and whole objects. Because each filter is reused at every position, CNNs need far fewer parameters than dense networks and recognise a pattern wherever it appears.

Like an inspector who checks every tray with the same small magnifying glass, square by square.

A piece of work that can pause partway through and resume later.

Calling an async function does not run it; it returns a coroutine. The work happens when the coroutine is awaited or handed to the event loop. A coroutine that is never awaited never runs, and Python warns about it.

Like a tray prepared and waiting by the oven: nothing bakes until someone puts it in.

A number that says how different two embeddings are in meaning: 0 means the same direction, larger means less related.

Cosine distance is one minus cosine similarity, which compares the angle between two vectors and ignores their length. Many vector databases can use it when searching, and with it, one minus the distance gives an easy-to-read similarity score. Other distance measures exist, so check which one your collection uses before interpreting scores.

Like comparing which way two arrows point, regardless of how long they are.

A score from −1 to 1 that says how closely two vectors point in the same direction.

Cosine similarity is the dot product of two vectors divided by the product of their lengths. It measures the angle between them and ignores how long they are: 1 means the same direction, 0 means unrelated, −1 means opposite. It is the most common way to compare text embeddings. The score is not a percentage, and what counts as "close" depends on the model.

Like checking whether two people are pointing at the same thing, regardless of how long their arms are.

What it costs an AI system to get one useful job done, including retries and failures.

Cost per task adds up the input and output tokens of every model call a task needs, multiplies by price, adds tool and search costs, and accounts for retries. Dividing by the success rate gives the cost per successful task. It is the number to compare with what the task is worth.

Like pricing a cake by everything that goes into one sellable cake, including the ones that collapsed in the oven.

An open-source Python framework where agents with roles work through a list of tasks as a crew.

You define agents with a role, goal and backstory, tasks with a description and expected output, and a crew that runs them sequentially or under a manager agent. It reads clearly, even to non-engineers. It adds its own prompt text to every call, which costs tokens and can make behaviour harder to debug.

Like a film crew with job titles and a call sheet listing who does what, in what order.

The way your standards for "good output" change as you look at more outputs.

People often discover what they want only by seeing what they get, such as realising they dislike emoji in replies. Criteria drift means eval definitions and labels need updating over time, and judges need re-aligning when they do.

Like a baker who only decides how sweet the icing should be after tasting a few batches.

A model that reads a question and a passage together and outputs one relevance score.

Because it sees both texts at once, a cross-encoder judges relevance much more accurately than comparing separate embeddings. It cannot be precomputed and runs once per question-passage pair, so it is used to rerank a short candidate list, not to search a whole collection. Its scores are raw numbers to compare with each other, not probabilities.

Like a proper interview after a quick CV screen.

The four basic things you do with stored data: create, read, update and delete.

Every database supports CRUD operations. In vector databases they appear as add or upsert, get or fetch, update, and delete, alongside the similarity search that makes them special. Handling update and delete properly is what keeps search results current.

Like the four things you do with folders in a cabinet: file new ones, look at them, change them and throw them away.

A loss that trains a model to output a text sequence without knowing which image column belongs to which character.

The model outputs a probability for every character, plus a special blank, at each horizontal position. CTC sums over every way of aligning the target text to those positions, where repeated characters merge and blanks are removed. Decoding takes the most likely symbol at each position, collapses repeats and drops blanks.

Like transcribing a song from a recording where you know the lyrics but not when each word starts.

A terminal program that sends HTTP requests and shows the responses.

curl is the quickest way to test an API without writing code: `curl -i URL` shows the status, headers and body. API documentation often gives curl examples. Use environment variables like $ANTHROPIC_API_KEY instead of pasting keys, so they stay out of your shell history.

Like phoning a supplier directly to check something, instead of filling in the full order system.

Your own named error type, made by creating a class that inherits from `Exception`.

Custom exceptions like `OrderNotFoundError` or `LLMResponseError` speak your app's language and can carry useful data, like the order ID. A shared base class lets callers catch a whole family at once. `raise NewError(...) from error` keeps the original cause attached.

Like having specific labels for kitchen problems, "out of stock" or "oven down", instead of shouting "problem!".

A function you write and describe so an agent can use it, such as looking up an order or searching a handbook.

A good custom tool has a clear name, typed inputs, a description that says what it returns and when to use it, small outputs, and helpful error messages. In smolagents you create one with the @tool decorator or by subclassing Tool when it needs setup or state. Tool design often matters more than prompt wording.

Like a well-labelled kitchen gadget: anyone can pick it up and know what it does.

A query language for graphs where you draw the pattern you are looking for, like (c:Customer)-[:PLACED]->(o).

Cypher uses round brackets for nodes and arrows for relationships. MATCH finds a pattern, WHERE filters it, RETURN chooses the output, CREATE adds data and MERGE finds-or-creates it. It began with Neo4j, is used by other graph databases, and inspired the ISO GQL standard.

Like sketching the shape of the answer on a whiteboard and asking the database to find every place it appears.

Creating varied training examples by randomly flipping, cropping, rotating or recolouring the originals.

Augmentation teaches a model that a croissant photographed slightly to the left or in warmer light is still a croissant, which reduces overfitting. It is applied to training data only, freshly at random each epoch. Augmentations must not change the correct label, so a flip suits pastries but not digits like 6 and 9.

Like training a new staff member with photos of the same pastry taken from many angles and in different lighting.

A LlamaIndex component that reads a source, such as a folder, Notion or Google Drive, and returns documents.

SimpleDirectoryReader handles a folder of mixed files; many more connectors are available as separate packages through LlamaHub. They play the same role as LangChain's document loaders. Connectors are where frameworks save the most time, because other people have already handled each service's quirks.

Like a set of plug adaptors that let one charger work in any country.

When the inputs a model sees in production start to look different from the data it was trained on.

Data drift happens when the world changes (a new office crowd near a store) or when something upstream breaks (a temperature feed switching units). Because it shows up in the inputs, you can detect it before the real answers arrive, using statistical comparisons such as the Kolmogorov–Smirnov test. Alert on the size of the change, not only on statistical significance.

Like a chef trained on summer produce suddenly cooking with winter vegetables: the recipes stay the same, but the ingredients have changed.

Secretly moving private data out of a system to somewhere an attacker can read it.

In AI apps, exfiltration channels include email or messaging tools, web requests, links the model writes, and Markdown images a browser loads automatically. Removing or tightly restricting outbound channels is one of the strongest defences against prompt injection.

Like smuggling the customer list out of the shop inside an ordinary-looking parcel.

When information the model would not have at prediction time sneaks into training or evaluation.

Common causes are shuffling time series before splitting, computing normalisation statistics on all data, and using features that are only known afterwards. Leakage makes test scores look better than real-world performance. Split by time, compute statistics on training data only, and ask of every feature: would I know this when predicting?

Like an exam where some answers are printed inside other questions.

Corrupting the data a model learns from or retrieves from, so it misbehaves in chosen situations.

An attacker might slip harmful examples into fine-tuning data or plant misleading documents in a search index. The model then learns or repeats the attacker's pattern. Defences include knowing where data comes from, reviewing and filtering it, keeping untrusted content out of training sets and running evals before and after retraining.

Like someone quietly swapping a few recipe cards in the archive so a cake comes out wrong months later.

Rules or promises about which country or region your data is physically stored in.

Laws, contracts and customers sometimes require data to stay in a particular country. That can rule a cloud service in or out depending on its available regions. Check residency requirements before choosing a database or loading any personal data.

Like a rule that the family jewellery must stay in a bank locker in your home city.

How long, and for what purposes, a provider keeps the data you send to its API.

Before sending real customer data, check whether the provider trains on it, how long it keeps requests (often a limited period for abuse monitoring), and where it stores them. Free tiers can have looser terms than paid ones. When data must not leave your premises, self-hosted open models are the alternative.

Like asking a courier how long they keep copies of the letters they deliver, and who can read them.

The kind of value something is, such as text, a whole number or true/false.

Every Python value has a type, and the type decides what you can do with it. The everyday ones are `str` (text), `int` (whole numbers), `float` (decimals), `bool` (`True` or `False`) and `None`. Python will not silently mix types, so `"3" + 4` is an error.

Like measuring flour by weight and eggs by count: you cannot add a kilo to a dozen.

Software that stores data in an organised way and answers questions about it quickly and safely.

A relational database keeps data in tables of rows and columns linked by keys, applies many simultaneous writes without losing any, and answers queries written in SQL. AI apps use one for users, conversation history, business data behind tools, cost logs and eval results.

Like a set of ledgers with a bookkeeper who files every entry correctly and answers any question about them.

A Python shortcut for classes that mainly hold data.

Adding `@dataclass` above a class with typed fields writes the constructor, a readable printout and an equality check for you. Use `field(default_factory=list)` for list defaults and `frozen=True` for read-only records. Dataclasses do not check value types at runtime; Pydantic does.

Like a pre-printed order form: the boxes are already drawn, you fill them in.

A table of data with named columns and labelled rows, used by the pandas library.

Each column is a Series, backed by a NumPy array with one data type. Check `shape`, `head()`, `dtypes` and `isna().sum()` first, then use masks, `groupby` and `pivot_table` instead of looping over rows.

Like the order spreadsheet, where each column is its own egg tray.

The PyTorch object that shuffles a dataset and serves it in batches.

A DataLoader wraps a Dataset, which returns one example at a time, and handles batching, shuffling, the last partial batch and loading in parallel worker processes. Shuffle training data; do not shuffle validation or test data.

Like a kitchen runner who fetches items from the storeroom and brings them to the station on trays.

A collection of examples used to train or test a model.

For a delivery-time model, a dataset is past deliveries with their distances and actual times. A model learns whatever is in its dataset, including mistakes and oddities, and is least reliable on inputs unlike anything in it. Datasets are also used to test models, which is the basis of evals.

Like the stack of past orders a new manager studies to learn the shop's rhythms.

The SQL statements, such as CREATE TABLE, that define a database's tables and columns.

Giving a model the DDL for the relevant tables, pulled from the database itself, stops it inventing table and column names. DDL shows structure but not business meaning, so add definitions like what counts as a late delivery.

Like the labelled floor plan of a warehouse, showing which shelves exist but not what the stock is for.

A tool that pauses a running program so you can inspect its values step by step.

Put `breakpoint()` on a line and Python pauses there, opening the built-in debugger `pdb`. You can print variables, run one line at a time and continue. Code editors offer the same with clickable breakpoints.

Like pausing a cooking video to look closely at exactly what is in the pan.

The second phase of answering a request, where the model writes the reply one token at a time.

Each new token depends on the ones before it, so decode is sequential: one full pass through the model per token. This is where most of the time and cost of a request goes, which is why output tokens are slower and pricier than input tokens. Providers batch many users together to make decode efficient.

Like adding ingredients one at a time, waiting for each to mix before deciding the next.

The half of a transformer that writes text one token at a time, only looking at what came before.

Decoders use masked (causal) attention. In the original encoder-decoder design they also had cross-attention to read the encoder's output. Decoder-only models such as GPT and Llama are the basis of almost every chat model today.

Like a storyteller who writes one word after another and never peeks ahead.

An agent built to work through long, complex tasks by planning, delegating to subagents and keeping notes in files.

Deep agents add four ingredients to the basic agent loop: an explicit plan, subagents with clean contexts, a file-system scratchpad and detailed instructions. Together these keep the goal visible and each context window small, which lets the agent stay on track across dozens or hundreds of steps. Claude Code is a well-known example.

Like an editor running an investigations desk rather than a single reporter working alone.

Machine learning that uses neural networks with many layers, good at messy data like photos, sound and text.

Each layer of a deep network learns to notice something slightly more abstract than the layer before, such as edges, then shapes, then objects. Deep learning took off around 2012 thanks to large datasets and GPUs, and it powers image recognition, speech recognition and today's language models. It usually needs much more data and computing power than classic ML.

Like an assembly line of inspectors, each one looking for a slightly bigger pattern than the last.

A deep agent specialised in researching a question across many sources and producing a cited report.

Deep research agents clarify the question, plan sub-questions, send subagents to search and read in parallel, then synthesise and verify a report with sources. They can take many minutes and use far more tokens than a normal chat, so they suit tasks worth an analyst's afternoon. Several AI assistants offer this as a built-in feature.

Like hiring a small research team for a day instead of asking one person a quick question.

Stacking several independent protections so that what slips past one is caught by another.

No single check stops every attack, so AI systems combine input checks, a model with minimal power, a tool gateway in code, output checks, human approval and monitoring. The layers should have different weaknesses; at least one strong layer should be deterministic code that a fooled model cannot talk its way past.

Like a shop that uses a till limit, a two-key safe, a camera and a nightly cash count, none perfect alone.

A marker, such as a tag or a line of symbols, that fences off one part of a prompt from another.

Delimiters like <email>...</email>, triple quotes or ### lines show the model where your instructions end and the input data begins. They let you refer to sections by name and make it harder for text inside the data to be mistaken for instructions. They help against prompt injection but are not a complete defence.

Like the plastic divider on a supermarket belt that separates your shopping from the next person's.

Passing a function the things it depends on, such as a client, instead of creating them inside it.

If a function receives its model client as an argument, a test can pass in a fake one. If the function creates the client itself, faking it is awkward. This small design choice makes LLM code much easier to test.

Like handing a baker the oven to use, so you can swap in a test oven when checking the recipe.

To take the next job off a queue so it can be processed.

Workers dequeue jobs, mark them as started and run them. RQ workers use a blocking pop, waiting on the queue server without using CPU until a job appears. If a worker dies mid-job, the job is eventually detected as abandoned and moved to the failed registry.

Like a cook pulling the next order slip off the rail.

How much an output changes when you nudge an input a tiny bit, and in which direction.

A derivative is the slope of a function at one point. A positive slope means increasing the input increases the output; a negative slope means the opposite. A partial derivative is the slope with respect to one input while holding the others still, which is how each parameter’s effect on the loss is measured.

Like feeling whether the ground slopes up or down under your feet in the fog.

A collection that maps names (keys) to values, so you can look things up by name.

Written with curly braces, like `{"croissant": 85}`. Keys are unique, lookups are fast, and `.get(key, default)` avoids crashing on missing keys. JSON objects become dictionaries in Python.

Like the menu board: find the item name, read its price.

A listing of only the lines removed and added between two versions of a file.

Lines starting with - are removed and lines starting with + are added. Asking a model for a diff makes changes small to review and guarantees everything else is untouched. Tools like git apply --check refuse a diff that does not match the real file.

Like tracked changes in a document, showing only what was edited.

A model that creates images by starting from random noise and removing it step by step, guided by a text prompt.

During training, noise is added to real images and the model learns to predict it. To generate, it starts from noise and repeatedly removes the predicted noise, steered by the prompt, usually in a compressed latent space. Classic models take 25 to 50 steps; turbo variants take 1 to 4. Diffusion models often misspell text and can get cultural details wrong.

Like a sculptor who starts with a rough block and chips away a little at a time until the shape you described appears.

Squashing many numbers per item down to a few, usually two or three, so you can plot or compress them.

Embeddings have hundreds of dimensions, which you cannot draw. Dimensionality reduction methods such as PCA and UMAP produce two-dimensional versions that keep some of the original structure. Every method throws information away, so pictures should be confirmed with real measurements in the full space.

Like casting a shadow of a 3-D object on a wall: you choose the angle so the shadow keeps the shape you care about.

The half of a GAN that looks at an image and guesses how likely it is to be real rather than generated.

It is trained like an ordinary yes-or-no classifier on a mix of real and generated images. Its judgements are the only training signal the generator gets. If it becomes too strong too quickly, the generator's feedback weakens and training stalls.

Like a head taster trying to spot the apprentice's sweets on an unlabelled tray.

Training a small "student" model to imitate a large "teacher" model.

For classifiers, the student matches the teacher's softened probabilities, which carry information about near-miss answers, blended with true labels. For language models, it often means fine-tuning a small model on text the big model wrote. The student can approach the teacher's skill on the targeted tasks at a fraction of the cost, but cannot exceed its knowledge.

Like an apprentice learning by watching a master chef's every judgement, not only the final dishes.

A tool that packages software with everything it needs so it runs the same way on any machine.

Docker builds images and runs them as isolated containers. It ends "it works on my machine" problems and makes it easy to run services like Redis, Postgres, Ollama and Milvus without installing them by hand. On macOS and Windows you use Docker Desktop; on Linux, Docker Engine.

Like shipping a sealed, ready-to-plug-in kitchen instead of a recipe and hoping the other kitchen matches.

A way to describe and start several containers together from one file.

A compose.yaml lists services (your app, Redis, a database), their images or build folders, ports, volumes and environment. `docker compose up -d` starts them all and `docker compose down` stops them. Services reach each other by service name.

Like a floor plan for a whole bakery: ovens, fridges and counters placed and connected in one go.

A sealed, read-only package of an operating system's files, a runtime, libraries, code and a start command.

Images are built in layers from a Dockerfile, named with a tag like redis:7, and stored in registries such as Docker Hub. One image can start any number of containers. Never put secrets inside an image.

Like a batch of frozen dough: fully prepared, ready to be baked as many times as you like.

Storage managed by Docker that survives when a container is deleted.

Anything written inside a container disappears with it. A named volume, mounted with -v name:/path, keeps those files on the host disk and plugs them into each new container. For local LLMs this keeps multi-gigabyte models and app data, such as Open WebUI's chats, across upgrades.

Like a locker at the station: you can swap suitcases as often as you like, and what is in the locker stays put.

A text file of step-by-step instructions Docker follows to build an image for your app.

A Dockerfile starts FROM a base image, copies in your files, installs dependencies with RUN and sets the start command with CMD. Docker caches each step as a layer, so putting dependency installation before copying code makes rebuilds fast. Pair it with a .dockerignore that excludes .env.

Like the recipe card for making the frozen dough.

Code that reads a file or source, such as a PDF or web page, and returns its text with labels like page and file name.

Loaders turn many formats into one standard shape: text plus metadata. They are less reliable than they look: scanned pages come back empty, tables come back scrambled, and repeated footers sneak in. Always read a sample of a loader's output before indexing it.

Like a translator who turns letters in many languages into one language your team reads.

Answering questions about images of documents, such as the total on a photographed invoice.

One approach runs OCR to get words and their positions, then uses a layout-aware model like LayoutLM to point at the answer span. The newer approach gives the document image straight to a vision-language model. The first cannot invent text that is not on the page but depends on OCR quality; the second handles messy layouts better but can invent.

Like asking an accountant to find the total on a crumpled receipt.

Multiply two lists of numbers position by position, then add up the results.

The dot product of (3, 4) and (4, 3) is 3×4 + 4×3 = 24. It grows when two vectors point the same way and also when they are longer. When every vector has been normalised to length 1, the dot product equals cosine similarity, which makes it the cheapest way to score similarity.

Like scoring agreement by how loudly people shout "yes": direction matters, but so does volume, unless everyone speaks at the same volume.

A simpler alternative to RLHF that learns from preference pairs directly, with no separate reward model or reinforcement-learning loop.

Its loss raises the probability of the chosen answer and lowers that of the rejected one, each measured relative to a frozen reference model. It is stable and cheap to run, so it is widely used for open models.

Like learning straight from "this one, not that one" feedback instead of first building a scoring rubric.

The small, fast model that proposes tokens for a bigger model to verify in speculative decoding.

It must usually share the big model's tokenizer, and it should agree with the big model often, or its guesses are wasted work. Alternatives include copying spans from the prompt or small extra prediction heads attached to the big model.

Like the junior who drafts a reply for the manager to approve or correct.

A regularisation trick that randomly switches off some neurons on each training step.

Dropout forces the network to spread knowledge across many neurons instead of relying on a few, which reduces overfitting. It is active only in training mode (model.train()) and switched off for evaluation and production (model.eval()). A typical rate is 0.1 to 0.5.

Like a kitchen where a random third of the staff are off each day, so everyone learns every station.

Stop training when the validation score stops improving, and keep the best version.

Track validation loss each epoch, save the model whenever it reaches a new best, and stop after a set number of epochs without improvement (the patience). Then reload the best weights. It is the simplest and often most effective defence against overfitting.

Like a coach who stops the drills the moment mock-exam scores stop rising.

A fixed arrow that says which step runs after another.

In LangGraph, add_edge("a", "b") means node b always runs after node a. The special names START and END mark where a run begins and finishes. Edges that need a decision are conditional edges instead.

Like the arrow on a kitchen chart saying that baking always goes to decorating.

An activation like ReLU for positive values but with a smooth, gently negative curve for negative ones.

ELU outputs x for positive x and α(eˣ − 1) for negative x, levelling off at −α. Negative inputs still produce a small output and gradient, so neurons do not die, and average activations stay closer to zero. It costs a little more to compute than ReLU.

Like a dimmer switch that never quite turns the light fully off.

A list of numbers that captures what a piece of text means, so similar meanings get similar numbers.

An embedding model turns text (or images) into a vector of hundreds or thousands of numbers. Texts about similar things land close together, even when they share no words, which makes search by meaning possible. Embeddings are the subject of Module 07.

Like a GPS coordinate for meaning: "cashew allergy" and "contains tree nuts" land close together on the map.

Unfair or skewed patterns that an embedding model picked up from the text it learned from.

Embedding models absorb associations from human writing, such as linking some jobs more closely with one gender, or treating text in one language differently from the same meaning in another. Systems built on the embeddings inherit these patterns. You find bias by probing the space and measuring results separately for each group.

Like a new employee who learned the job by reading old company emails and picked up the old prejudices along with the useful habits.

A model whose only job is to turn a piece of text into a fixed-length list of numbers that captures its meaning.

An embedding model reads text and returns one vector, such as 384 or 1,536 numbers, no matter how long the input is. It is trained so that texts with similar meanings get nearby vectors. Examples include OpenAI text-embedding-3-small and the free all-MiniLM-L6-v2. Vectors from different embedding models cannot be compared with each other.

Like a librarian who files every book on a giant map so that books on similar subjects end up on neighbouring shelves.

The half of a transformer that reads a whole text at once, looking both left and right, to understand it.

Encoders use attention with no causal mask, so every token sees every other token. They output one rich vector per token, which is ideal for classification, search embeddings and tagging. BERT is the best-known encoder-only model.

Like a crossword solver who uses the letters on both sides of a gap.

One specific thing a web API offers, identified by an HTTP method and a path.

GET /orders/CR-1042 and POST /triage are two endpoints. Each has a contract: what the request must contain, what the response looks like, and which status codes it can return. Other programs, in any language, call endpoints over HTTP.

Like one item on a restaurant menu, with its own order slip.

Working out which mentions refer to the same real thing, and which only look alike.

"Groundnuts", "Peanut" and "moongphali" should become one allergen node; two different customers called Rohan must stay two nodes. Entity resolution uses session identity, alias tables, catalogue matching and human review for borderline cases. It is the hardest part of building graph memory from conversations.

Like a library's authority file, which files "R. Tagore" and "Rabindranath Tagore" under one author.

A named value, like ANTHROPIC_API_KEY, that programs started from your shell can read.

Environment variables pass settings and secrets to programs without writing them into code. Set them with `export NAME=value` (or `$env:NAME = "value"` in PowerShell), or load them from a .env file. They last only for that shell session unless saved in a startup file, and SDKs read API keys from them.

Like notes pinned on the kitchen noticeboard that every cook on the shift can read.

Memories of specific past events, each with a time and what happened.

Episodic memories record experiences such as "28 October: hamper arrived late, ₹200 voucher given". They help an agent stay consistent with past decisions and notice patterns. Episodes usually matter for months rather than forever, so they are good candidates for expiry or consolidation.

Like a diary entry: on this day, this happened.

One complete pass through all the training data.

Training usually runs for several epochs. The number of update steps is the number of mini-batches per epoch times the number of epochs. Validation is typically checked once per epoch to decide when to stop.

Like reading the whole recipe book once from cover to cover.

Reading many real outputs of your AI app to discover how it actually fails, before measuring anything.

Error analysis means sampling 50 to 100 traces, writing a short note on what went wrong in each (open coding), grouping the notes into failure modes (axial coding) and counting them. The result tells you what to fix first and which evals are worth building. It is the "analyse" step of the analyse, measure, improve loop.

Like a shop owner standing in a struggling store for a day before deciding what to change.

Handing a conversation from an AI system to a person.

A good agent escalates when stakes are high (safety, money), when it cannot answer from its sources, when the customer asks for a person, or when things are going badly. The rules live in code, and the handoff includes a summary so the customer never has to repeat themselves.

Like a shop assistant calling the manager over and briefing them before they arrive.

The straight-line distance between two points, as a ruler would measure it.

Euclidean distance is the square root of the sum of squared differences between two vectors. Smaller means more similar. For vectors normalised to length 1, squared distance equals 2 − 2 × cosine similarity, so ranking by distance and by cosine gives the same order. Some vector databases use it as their default metric.

Like measuring how far apart two pins are on a map with a ruler.

A repeatable test that measures how good an AI system's answers are across many examples.

An eval runs your system on a set of example inputs and scores the outputs, with exact checks, rules or another model acting as judge. Evals turn "it seemed fine" into a number, catch regressions when you change a prompt, and are one of the most important skills in AI engineering. Module 23 covers them.

Like a tasting panel that samples every batch against the standard instead of trusting that the oven "seemed fine today".

An automatic rule that blocks a change from shipping unless it passes agreed evaluation thresholds.

An eval gate runs the eval set against a proposed change and compares the results with the current version. Good gates check each important category separately ("allergy cases must stay at 100%") as well as the overall score, because an average can rise while a critical category collapses. In practice the gate runs in CI on every pull request.

Like a bakery’s quality check that no tray passes until it meets the weight and taste standards, however good the tray looks.

A loop where one model call produces work, another critiques it against clear criteria, and the first revises.

It works when you have specific evaluation criteria and revisions measurably help, as with copywriting or translation. It needs a round limit, and the evaluator must be checked against human judgement, because an evaluator that is too lenient or too harsh makes the loop useless.

Like a writer and an editor passing a draft back and forth until it is ready to print.

The part of async Python that runs coroutines and switches between them while they wait.

The event loop keeps track of which tasks are ready and which are waiting, running one at a time and switching at each `await`. `asyncio.run(main())` starts it. A blocking call, like `time.sleep`, stops it from switching.

Like a head baker with a whiteboard of which oven is waiting for what.

New or changed data becomes visible to every reader after a short delay, not instantly.

Distributed systems often acknowledge a write before every part of the system has seen it. In Pinecone serverless, a freshly upserted record may take a few seconds to appear in query results. Tests that write and immediately read should wait or poll.

Like a notice pinned at head office that reaches every branch noticeboard by the end of the morning.

Changing a file by giving the exact text to find and the text to put in its place, rather than rewriting the whole file.

The tool refuses unless the old text appears exactly once, and its error tells the model to re-read the file or add surrounding lines. This keeps edits small and reviewable and stops the model from accidentally changing lines it had to retype from memory. Most coding agents, including Claude Code, edit files this way.

Like correcting one word in a printed recipe with correction tape instead of rewriting the whole card by hand.

Python's signal that something went wrong while the program was running.

An exception is an object, such as a `ValueError` or `KeyError`, raised where the problem happened. It travels up through the functions that called that code until something catches it; if nothing does, the program stops. Exceptions are classes in a family tree, so catching a parent catches its children.

Like a line cook shouting "out of saffron!" up the kitchen chain until someone who can fix it answers.

Giving an AI agent more power than its job needs, such as tools that can do anything.

Examples include a refund tool with no limit, an email tool that can write to anyone, or a general SQL tool. When the model is fooled or makes a mistake, excessive agency turns it into real damage. The fix is least privilege: narrow tools, limits in code and human approval for high-impact actions.

Like giving a trainee the keys to the safe on their first day.

When gradients grow enormously as they pass back through layers, causing wild updates or nan.

If the slopes multiplied during backpropagation are mostly above 1, the gradient grows exponentially. Training shows sudden loss spikes and then nan. Gradient clipping, careful initialisation, normalisation and lower learning rates are the usual fixes; recurrent networks are especially prone.

Like microphone feedback: a small sound loops through the speaker and becomes a deafening screech.

Retrying a failed request with waits that double each time.

Wait 1 second, then 2, then 4, up to a fixed number of attempts. Growing waits give a struggling server room to recover instead of hammering it. Only retry temporary failures, and be careful retrying requests that create or charge something.

Like calling back a busy shop after one minute, then two, then four, instead of redialling non-stop.

Answering a question by pointing at the exact words in a given text, rather than writing new ones.

An extractive QA model scores every token as a possible start and end of the answer, and picks the best span. Models trained on SQuAD 2.0 also score a special "no answer" position, so the gap between the best span and "no answer" works as an honest confidence. Because answers are always quotes, the model cannot invent facts, but it also cannot combine information or explain.

Like a student who answers every question by highlighting a sentence in the textbook.

A summary made by picking the most important sentences from the source, word for word.

Extractive summaries cannot invent anything because every sentence comes from the original, but they can read choppily. Asking a model for supporting quotes alongside its own summary gives you some of this safety.

Like highlighting the key sentences in a book with a marker.

Stable facts and preferences about a specific person or account, like an allergy or a delivery address.

Factual memories are short, precise and high value: "allergic to peanuts", "prefers eggless cakes", "lives in Kothrud". When a fact changes, the old version should be replaced rather than kept alongside it. Getting who a fact is about right matters as much as the fact itself.

Like the notes on a customer's loyalty card: who they are and what they need.

A specific, recurring way your AI app goes wrong, such as "reply contradicts the order tool".

Failure modes come out of error analysis. A good one is narrow enough that two people would classify the same trace the same way and clear enough to point at a fix; "hallucination" is usually too broad. Each important failure mode becomes one binary eval.

Like a fault code on a delivery van's dashboard: it names one problem, not "something is wrong".

A score for how much of an answer is supported by the passages that were retrieved.

The answer is broken into individual claims, and each is checked against the retrieved context. Faithfulness is the fraction that are supported. A low score means the model added or changed facts, which is the RAG form of hallucination; fix the prompt, abstention rules or model.

Like checking that every line in a news story can be traced to a source in the reporter's notes.

A backup option the system switches to automatically when the first choice fails.

In an LLM app, a fallback chain lists providers or models in order and moves to the next when one returns outage-type errors such as 429, 5xx or timeouts. The chain should end with a safe canned reply and a human handoff. Because a backup model behaves differently, every fallback needs its own evaluation.

Like calling the backup flour mill in Nashik when the Satara mill is on strike.

A Python web framework that turns functions into web API endpoints.

FastAPI uses type hints and Pydantic models to validate requests, convert parameters, shape responses and generate interactive documentation automatically. It supports async endpoints and streaming responses, which makes it a popular choice for serving LLM features.

Like a ready-built shop counter with a menu board and a slip checker included.

The easy way to build MCP servers in Python, where decorators turn functions into tools, resources and prompts.

FastMCP reads each function's name, type hints and docstring to build the tool's name, input schema and description. It ships in version 1 of the official MCP Python SDK as mcp.server.fastmcp.FastMCP; version 2 of the SDK renamed it MCPServer with the same decorators.

Like a label maker that prints a proper product label from the details you type in.

A setting your app reads while running that turns a feature on or off without deploying new code.

Flags can switch a feature for everyone, for a percentage of users, or for staff only. They should live in a shared store such as Redis or a flag service and be read on every request, so changes apply in seconds. Each flag needs a safe default for when the flag store itself is unavailable.

Like a light switch on the wall, instead of rewiring the house every time you want the lights off.

Combining the outputs of separate encoders for different kinds of input into one representation.

Multi-input models give each input type its own encoder (a CNN for images, an RNN for sequences, embeddings and an MLP for metadata) and then fuse the results, most simply by concatenating them. Late fusion combines encoded features; early fusion combines raw inputs. Other methods include adding, gating and attention.

Like a head chef combining reports from the pastry, bread and coffee stations into one plan.

The grid of outputs one convolution filter produces, showing where its pattern appears in the image.

Each filter in a convolution layer produces one feature map; a layer with 64 filters outputs 64 channels. Early feature maps highlight edges and colours; deeper ones respond to textures, parts and objects. Their height and width shrink as pooling and strides are applied.

Like a heat map of a tray showing where the flaky layers are.

Rescaling input features so they are on comparable scales, usually mean 0 and standard deviation 1.

Unscaled features let large-valued inputs dominate and make gradient descent unstable. Standardising computes (value − mean) / standard deviation using training data only. The same statistics must be saved and reused at prediction time, or the model receives inputs it was never trained on.

Like converting every judge’s score to the same scale before adding them up.

The small neural network inside each transformer block that processes every token on its own after attention.

It is two linear layers with an activation in between, usually expanding to about four times the model width and shrinking back. In a standard block it holds around two thirds of the weights, and researchers believe much of a model's stored factual knowledge lives there. Mixture-of-experts models replace it with many smaller expert networks.

Like each team member going back to their desk to think after a meeting where everyone shared notes.

A routine that turns real-world signals, like complaints and thumbs-downs, into improvements and new tests.

Signals include thumbs up or down, hand-offs to humans, repeated questions and a regular read of random conversations. The loop only works if someone reviews them and turns each confirmed failure into a permanent eval case, so the same mistake cannot silently return. Explicit feedback is sparse and biased, so treat it as a pointer rather than a score.

Like a restaurant reading its comment cards every week and changing the recipe when the same complaint keeps appearing.

Showing the model a few worked examples of the behaviour you want before the real request.

In a chat, you can add example user and assistant messages you wrote yourself; the model treats them as its own earlier replies and follows their style and format. It is one of the simplest ways to control tone, length and output shape.

Like showing a new hire two good customer replies before they write their first one.

Giving a model several worked examples in the prompt so it copies your pattern.

Usually three to ten deliberately chosen examples covering every label and the hard boundary cases. Few-shot prompting teaches house rules far better than long definitions. The examples cost tokens on every call and must be kept separate from your test set.

Like tagging a dozen reviews side by side with a new colleague before letting them work alone.

A score for image generators that compares generated images with real ones; lower is better.

Both sets of images are passed through a pretrained image network and their features summarised by an average and a spread; FID measures the distance between those summaries. It penalises both poor quality and poor variety. It needs thousands of images and is only comparable when computed the same way.

Like comparing two bakeries by the average taste and the range of their whole menus, not one cake each.

The address of a file or folder on your computer.

An absolute path starts from the top of the drive, such as /Users/arjun/code/main.py or C:\Users\arjun\code\main.py, and works from anywhere. A relative path, such as ../crumb-lab/main.py, starts from the working directory. `..` means the folder above and `~` means your home folder.

Like a postal address: either the full address from the country down, or directions from where you are standing.

Training an existing model a little more on your own examples so it picks up a style, format or narrow skill.

Fine-tuning changes a model's parameters, unlike prompting or RAG, which only change its input. It is good for consistent style or format and specialised tasks, but it is a poor way to teach facts that change, and it costs time and money. Module 33 covers when it is worth it and how to do it.

Like sending an experienced baker on a short course in your house style, rather than training a new baker from scratch.

A field in every response that says why the model stopped writing.

The common values mean: finished naturally, hit the max tokens limit (the reply is cut off), hit a stop sequence, or declined for safety. Names differ by provider, such as end_turn, stop, STOP, max_tokens, length and MAX_TOKENS. Check it on every call before using the text.

Like the note in a cake box saying "finished" or "ran out of time, icing incomplete".

A test that sometimes passes and sometimes fails without any change to the code.

Flaky tests teach people to ignore failures. With LLMs, they often come from asserting exact text or setting a pass-rate threshold inside normal run-to-run wobble. Fix them by asserting properties, adding cases and setting thresholds below normal variation; retry only network errors, never quality failures.

Like a smoke alarm that goes off at random: soon nobody reacts, even to real smoke.

A faster, memory-light way to compute exactly the same attention, by working in small tiles in the GPU's fastest memory.

Standard attention writes the full score grid to slow GPU memory and reads it back several times. Flash attention processes tiles of queries, keys and values on-chip and uses an online softmax with running maximums and totals, so the grid is never stored. The result is exact, and memory grows linearly with length.

Like doing every step for one tray at the counter instead of carrying ingredients back and forth to the cold store.

Searching by comparing the query with every stored vector, one by one.

A flat index has no shortcuts, so it always finds the true nearest neighbours: perfect recall. Its time grows in step with the number of vectors, which is fine for tens of thousands and too slow for many millions. It is also the yardstick you use to measure an approximate index’s recall.

Like reading every spine on every shelf to be certain you have found the best book.

How many bits a computer uses to store each number, which sets both its accuracy and its memory cost.

FP32 uses 4 bytes per number, FP16 and BF16 use 2, and FP8 uses 1. Most models are shipped in 16-bit, so memory in gigabytes is roughly twice the parameter count in billions. Training increasingly uses lower precision for speed, with care to keep sensitive parts accurate.

Like recording prices to the paisa, the rupee, or the nearest ten rupees.

The simplest reasonable forecast, which any model must beat to be worth using.

Naive predicts tomorrow equals today; seasonal naive predicts tomorrow equals the same day last week (or last year). Baselines are cheap, surprisingly strong and turn a model’s score into a decision: “12.6 against 23.6” means something, “12.6” alone does not.

Like checking a new recipe against the one the bakery already makes before changing the menu.

A column that points to another table's primary key, linking rows together.

In a conversations table, customer_id is a foreign key to customers.id. Storing the link instead of copying the customer's details means each fact lives in one place, and JOINs bring the tables back together when you query.

Like writing a customer number on an order slip instead of copying their whole address every time.

Deliberately removing or down-weighting memories that are outdated, temporary or no longer wanted.

Agents forget in several ways: superseding a changed fact (a new address replaces the old), expiring temporary memories after a set time, consolidating many small episodes into fewer facts, and erasing everything about a person on request. Without forgetting, memory stores grow noisier, slower and riskier.

Like clearing out old prescriptions from a medical file so the doctor sees only what is current.

A named, reusable block of code that takes inputs and gives back a result.

Defined with `def`, a function takes parameters and uses `return` to hand a value back to the caller. A function without `return` gives back `None`. Its docstring describes what it does, and LLMs read docstrings when deciding which tool to call.

Like a recipe card: written once, followed by anyone, with different ingredients each time.

One of your own functions made available for an agent's model to call.

An agent SDK reads the function's name, type hints and docstring to describe it to the model, then runs it in your process when the model asks. Function tools are where your own systems, data and business rules connect to the agent.

Like the shop's own order book, which only your staff keep and read.

A pair of networks, a forger and a detective, trained against each other until the forger's fakes look real.

The generator turns random noise into images; the discriminator scores images as real or fake. They train in turns, so each improves as the other does. GANs can produce sharp images but are notoriously unstable and prone to mode collapse.

Like an apprentice confectioner whose only feedback is a head taster trying to tell their sweets from the master's.

A model on the Hub that you can only download after accepting its terms on the website.

Publishers such as Google and Meta gate some models so that users agree to a licence before downloading. You accept the terms on the model page while logged in, and your code must use a token from that same account. Until then, downloads fail with a 401 or 403 error that mentions a gated repo.

Like a members-only section of the library: the books are there, but you sign the form at the desk first.

AI that creates new content, such as text, images, audio or code, instead of only sorting or predicting.

Generative models produce new output that did not exist before, based on patterns learned in training. Large language models generate text; other generative models make images, music or speech. Generative AI is a part of deep learning, and its output can be fluent but wrong, so it needs checking.

Like the difference between a food critic who rates dishes and a chef who invents new ones.

A model that produces brand-new examples that look like they came from its training data.

Instead of labelling or scoring inputs, a generative model creates outputs: new images, new text, new audio. Image generators usually start from random numbers and transform them into a picture. GANs, VAEs, diffusion models and LLMs are all generative models.

Like a baker who has tasted thousands of cakes and can now invent a new one in the same style.

The half of a GAN that turns random numbers into fake images.

It never sees real images directly. It learns entirely from the discriminator's verdicts on its fakes, usually by trying to maximise the probability the discriminator assigns to them being real.

Like a forger who only learns from a detective saying "caught you" or "fooled me".

A single-file format for storing quantised models, used by llama.cpp and Ollama.

A GGUF file holds the weights, tokenizer and settings together. Names like Q8_0, Q5_K_M and Q4_K_M describe the quantisation recipe, roughly 8, 5 and 4 bits per weight. An 8-billion-parameter model at Q4_K_M is about 5 GB.

Like a vacuum-packed meal kit with the recipe card inside the bag.

A tool that records save points of your project so you can see changes, undo mistakes and collaborate.

Git tracks every change to your files as a series of commits. You can compare any two versions, return to one that worked, experiment on branches, and merge work from several people line by line. It runs on your own computer; GitHub hosts copies online.

Like a bakery's production logbook, with a photo of every recipe card each day and the power to roll back.

A website that hosts git repositories online for backup, sharing and code review.

You push commits from your laptop to GitHub and pull other people's commits down. GitHub adds pull requests for reviewing changes, plus issues, automation and more. GitLab and Bitbucket offer similar services.

Like a shared head-office archive where every store's logbook is copied, read and signed off.

GitHub's built-in automation that runs your scripts when something happens in a repository, such as a new pull request.

A workflow file in .github/workflows says which events trigger it, what permissions its token has and which steps to run on a fresh virtual machine. Secrets like API keys are stored in the repository settings and passed in as environment variables.

Like a standing instruction to the shop: "whenever a delivery arrives, check it against the order".

A fixed list of test questions with correct reference answers, used to measure an AI system the same way every time.

A good golden dataset mixes easy lookups, multi-part answers, codes and names, messy real wording and questions that cannot be answered. Reference answers are written or checked by people who know the truth. Every real failure found in production should be added so it cannot quietly return.

Like a standard set of test bakes a new oven must pass before it goes into the shop.

A family of decoder-only transformers from OpenAI that generate text by predicting one token at a time.

GPT models are pretrained with causal language modelling. As they grew, they showed in-context learning: picking up new tasks from examples in the prompt. "GPT-style" now describes any decoder-only language model, including open families like Llama and Mistral.

Like a storyteller who has read a vast library and can continue almost any opening line.

A computer chip that does thousands of simple calculations at the same time, which is what AI models need.

GPUs were built for video games, but their many small parallel cores fit the multiply-and-add work of neural networks perfectly. Training a large model uses thousands of them for weeks; running one also needs GPUs, which is a big part of what model providers charge for.

Like a kitchen with a thousand small ovens instead of one big one: each is simple, but together they finish huge orders fast.

Giving a container access to the host computer's graphics card.

Containers cannot see the GPU unless it is handed through. With NVIDIA on Linux you install the driver and the NVIDIA Container Toolkit, then start the container with --gpus=all (or a deploy block in Compose); AMD uses the ROCm image with device flags. Docker on macOS cannot pass the Apple GPU through, so containers there run on the CPU.

Like running a gas line into a food stall: the stall works without it, but it can only cook on a hot plate.

Keeping an app useful, in a reduced way, when one of its parts fails.

Instead of showing an error, a degraded app might give a polite message and queue the question for a human, use a canned answer for common questions, or switch to a backup model. The right fallback depends on the cost of a wrong answer.

Like a shop taking cash and writing receipts by hand when the card machine breaks.

The list of slopes of the loss, one for every parameter, pointing in the direction where the loss rises fastest.

The gradient tells training which way each parameter should move: stepping against it lowers the loss. Frameworks like PyTorch compute it automatically with backpropagation. Its size also matters: tiny gradients mean slow learning, huge ones mean unstable training.

Like a compass that always points uphill, so you walk the other way to get down.

Capping the overall size of the gradient before each update to prevent explosive steps.

If the gradient’s total size exceeds a limit (often 1 to 5), it is scaled down to that limit while keeping its direction. In PyTorch this is torch.nn.utils.clip_grad_norm_ between backward() and step(). It is standard for recurrent networks and transformers.

Like a speed limiter on a delivery scooter: it can still go anywhere, just not dangerously fast.

The method training uses to decide which way, and how far, to turn each parameter to reduce the loss.

For each parameter, gradient descent measures the slope of the loss (the gradient) and moves the parameter a small step downhill. The step size is the learning rate: too small and training crawls, too large and it overshoots and blows up. You will learn the calculus behind it in Module 27.

Like walking down a hillside blindfolded, feeling which way the ground slopes and taking a small step that way.

TensorFlow’s tool for recording calculations so it can compute gradients.

Operations inside a with tf.GradientTape() block are recorded; tape.gradient(loss, variables) then returns the gradients. It is TensorFlow’s equivalent of PyTorch’s autograd and is the basis of custom training loops. A tape computes fresh gradients each time, so nothing accumulates between steps.

Like pressing record on a camera only for the part of the recipe you want to replay backwards.

A Python library for building quick web demos of models, including ready-made chat windows.

With gr.ChatInterface you write one function that takes the new message and the history and returns or yields the reply, and Gradio provides the whole chat UI. It is widely used for model demos, including on Hugging Face Spaces.

Like a ready-made tasting kiosk where you only plug in the recipe.

A database designed to store nodes and relationships and follow them quickly.

Graph databases keep direct links from each node to its neighbours, so a multi-hop query reads short lists instead of scanning whole tables. Most use a query language like Cypher. Neo4j and Kuzu are two examples, one a server and one embedded.

Like a city map with every road already drawn, rather than a list of addresses you must search each time.

Agent memory stored as nodes and relationships, so the agent can answer by following connections.

Instead of only saving memories as separate sentences, an agent with graph memory stores who relates to what: family members, allergies, purchases, preferences. It is usually combined with vector memory, which finds fuzzy starting points, while the graph handles exact multi-hop questions.

Like a family tree and order history kept alongside your diary, instead of only the diary.

Always choosing the single most likely next token.

Greedy decoding is what temperature 0 means. It makes output predictable, which suits extraction and classification, but long text can become flat or repetitive. Even greedy output is not perfectly repeatable on shared GPUs, because tiny numerical differences can break near-ties.

Like a baker who always follows the recipe to the gram and never improvises.

Giving the model the real facts it needs inside the request, so it answers from them rather than from memory.

A model asked about something it has never seen produces plausible text; given the order record, policy or document in the request, it answers accurately. Grounding is usually paired with an instruction for what to do when the facts do not cover a question. Retrieval (RAG) is grounding done automatically.

Like handing the new hire the order sheet before they answer the customer.

A multi-agent setup where agents take turns adding messages to one shared conversation.

Rules decide who speaks next, such as round-robin or a model choosing the next speaker, and termination conditions decide when it ends. Every agent reads the whole history on its turn, so token costs grow quickly. Without an owner and a stopping rule, group chats drift.

Like a committee meeting that only ends when someone calls the vote, or the clock runs out.

An attention design where several query heads share one set of keys and values, shrinking the KV cache.

A model might have 32 query heads but only 8 key-value heads, each shared by four query heads, which cuts the cache to a quarter with little quality loss. It is common in modern open models such as Llama 3. Multi-head latent attention pushes the same goal further.

Like four chefs sharing one well-organised pantry instead of each keeping their own.

A reinforcement-learning method that judges each sampled answer against the average of a group of answers to the same prompt.

Introduced by DeepSeek, it drops the separate critic model that PPO needs, which saves memory. Each answer's advantage is its reward minus the group mean, divided by the group spread. Combined with automatically checkable rewards, it trained DeepSeek-R1's reasoning.

Like grading each attempt on a curve against the student's own other attempts at the same question.

A simpler gated recurrent network with two gates instead of the LSTM’s three.

A GRU merges the LSTM’s cell and hidden state and uses an update gate and a reset gate. It has about three quarters of an LSTM’s parameters, trains a little faster and performs similarly on most tasks. Try both when building a recurrent model.

Like an LSTM’s binder with the eraser and pen combined into one tool.

Any mechanism that keeps an AI system's behaviour within safe, intended limits.

Guardrails come in layers: rules in the system prompt, checks in code on what goes in and what comes out, separate safety models, and humans reviewing risky actions. Prompt-level rules are the cheapest first layer but can be bypassed, so anything involving money, safety or privacy needs a layer outside the model too.

Like the rails on a mountain road: they keep most traffic safe, and you still want brakes and a careful driver.

The gap between you and what is really happening in your app's inputs and outputs.

You cannot read every message and reply, so your beliefs about your traffic drift from reality. Symptoms include production surprises and test sets that do not resemble real use. You cross it with error analysis and by rebuilding test sets from real data.

Like a founder who assumes customers mostly order cakes, when half the calls are about deliveries.

The gap between clear instructions and the model applying them reliably to every kind of real input.

Even with a clear prompt, a model can fail on particular shapes of input, such as long, mixed-language or multi-part messages. This is where evals earn their keep: measure the failure, then try fixes such as splitting the task, better retrieval, examples, validation or a stronger model.

Like a well-briefed trainee who still gets flustered when a customer asks about two orders at once.

The gap between what you want the model to do and what you actually told it.

The model knows only what is in its prompt and context, so unstated policies, formats and edge cases get filled with reasonable guesses. Fix specification failures by writing the missing instruction, before building any eval for them.

Like a trainee who gives full refunds because nobody mentioned the new store-credit rule.

When a model gives a fluent, confident answer that is false or made up.

Because a model generates likely-sounding text rather than looking facts up, it can invent policies, products or numbers. Giving it real facts (RAG), clear instructions and tools makes hallucinations much rarer, and evals measure how often they still happen. Nothing makes them impossible.

Like a nervous new employee who would rather invent an answer than say "I don't know".

Passing a conversation from one agent to another, which then takes over completely.

A triage agent might hand an allergy question to an allergy specialist that talks to the customer directly. Unlike delegation, the first agent steps back rather than waiting for a memo. Handoffs suit customer-facing routing and are built into several agent SDKs.

Like a receptionist transferring your call to the right department and hanging up.

A small endpoint a platform calls to decide whether your app is ready for traffic and still alive.

Platforms often separate a startup check (is it ready yet?) from a liveness check (has it frozen?). Health checks must be cheap and should not depend on outside services such as the LLM provider, or an outage elsewhere becomes a restart loop and a big bill. Whether dependencies are healthy is a job for monitoring.

Like a nurse checking a patient’s pulse: quick, frequent, and not a full medical exam.

The list of numbers a model builds internally to describe each token in its context.

Inside a transformer, every token is represented by a vector, for example 768 numbers, that is refined layer by layer. The final layer's vectors are the hidden states a head reads. Averaging them over a sentence is a simple way to get an embedding.

Like a chef's private tasting notes on each ingredient, written before deciding what dish to make.

Removing older messages from a conversation so each request stays within a token budget.

Good trimming removes whole user-and-assistant pairs from the oldest end, in chunks rather than one message per turn, and only when the history passes a generous budget. What is dropped is often folded into a summary, and critical facts are pinned so they survive.

Like moving old pages of a thick customer folder into an archive, keeping a summary sheet on top.

The share of test queries where the right answer appears somewhere in the top k results.

To measure search, you collect real queries with a known correct answer and check whether that answer shows up in the top k. A hit rate at 3 of 80% means four in five queries had the right item in the first three results. It turns "the demo looked good" into a number you can improve.

Like checking, for each of thirty customers, whether the cake they wanted was among the first three the assistant showed them.

A fast vector index that links each vector to its neighbours in a layered graph and searches by hopping.

HNSW builds a graph where every vector is linked to a few near neighbours, with sparser upper layers that act like motorways. A search starts at the top, hops toward the query, and drops down layer by layer. Settings M and ef_construction shape the graph; ef_search trades recall for speed at query time. It is the default index in many vector databases and needs plenty of memory.

Like crossing a city by motorway, then main roads, then local lanes, instead of driving down every street.

Labelled examples you keep aside and measure only once, at the end, for an honest score.

If you tune a prompt or model while looking at certain examples, it gets fitted to them and scores well on them regardless of real ability. A hold-out set is never used for tuning, so its score estimates how the system will do on new data. Use a separate development set for iteration.

Like a sealed exam paper that a student never sees while revising.

Handling more load by running more copies of the same component, such as more workers.

With a shared queue, extra workers on any machine pull from the same jobs, so capacity grows by adding copies. It also adds resilience, since one machine can fail without stopping the service. Limits come from memory per copy and from shared downstream services such as an LLM provider's rate limit.

Like hiring more bakers instead of buying a bigger oven.

A tool that runs on the model provider's own servers, such as built-in web search or file search.

You add a hosted tool to an agent with one line and the provider executes it, returning the result into the model's answer. It saves building and running the tool yourself, but ties you to that provider, adds per-use costs and sends your data through their service.

Like the bank's card machine on the shop counter: it works, but it belongs to someone else.

The set of rules programs use to send requests and responses over the web.

An HTTP request has a method (such as GET or POST), a URL, headers and an optional body. The response has a status code, headers and a body. Browsers, apps and model APIs all speak HTTP; HTTPS is the encrypted version.

Like a postal system with standard order slips and reply slips that every business understands.

A name and value attached to an HTTP request or response that describes the message.

Headers carry details such as the body format (content-type), your API key (x-api-key or Authorization), the API version, or how long to wait before retrying (retry-after). Never log authorisation headers, or your keys end up in log files.

Like the details written on an envelope: sender, account number, language of the letter inside.

A three-digit number on every web response that says how the request went.

2xx codes mean success (200 OK, 201 Created). 4xx codes mean the request has a problem (400 bad request, 401 bad key, 404 not found, 422 invalid input, 429 too many requests). 5xx codes mean the server had a problem (500, 502, 503). Using the right code lets other programs decide automatically whether to fix, retry or give up.

Like the coloured stamp on a returned letter: delivered, wrong address, or post office closed.

A public website where people share ready-trained AI models, datasets and demo apps for anyone to download or try.

The Hub hosts hundreds of thousands of models and datasets, each stored in a versioned Git repository named owner/name. You can browse them, read their model cards, download them with the hf CLI or Python libraries, or call some of them over the internet through Inference Providers. Spaces are small demo apps hosted alongside.

Like a public library crossed with GitHub: borrow any book, see every past edition, and check the rules printed inside the cover.

A design where people review or make the decisions an AI system should not make alone.

The AI handles routine work and prepares the rest, while humans approve risky actions, handle escalations and check samples. It trades some automation for safety and trust, and is standard practice for anything involving money, health or legal risk.

Like an autopilot that flies the cruise while the pilot handles take-off, landing and anything unusual.

Running keyword search and meaning-based vector search together and combining their results.

Keyword search catches exact codes, names and rare words; vector search catches paraphrases and synonyms. Hybrid search runs both and merges the rankings, usually with reciprocal rank fusion, so the right chunk lands in the top handful for a wider range of questions. A reranker then picks the best few.

Like asking both the catalogue and an experienced librarian, then shortlisting the books both recommend.

A retrieval trick where an LLM writes a fake answer passage and you search with that passage instead of the question.

Short, casual questions look different from long, formal documents. A hypothetical passage written in the documents' style lands closer to the real passages in embedding space. It may contain wrong details, which is fine because it is never shown to anyone; only the real retrieved chunks are used for the answer.

Like sketching what the dish should look like and showing the sketch to the chef, rather than describing it vaguely.

Work that is slow because it waits for the network or disk, not because the computer is busy.

A model API call spends almost all its time waiting for a remote server, while your own processor sits idle. I/O-bound work speeds up hugely with concurrency. CPU-bound work, like resizing thousands of photos, does not.

Like waiting for a courier: a faster kitchen at your end does not make the parcel arrive sooner.

The property that doing an operation twice has the same effect as doing it once.

Retries, double-clicks and webhook redeliveries all repeat requests. An idempotency key, such as a client-generated message ID used as the job ID, lets the server recognise repeats. Tasks with side effects, like sending a reply, should check whether the effect already happened before doing it again.

Like pressing a lift button twice: the lift still comes only once.

A unique ID sent with a request so the server can recognise repeats and avoid doing the same action twice.

The caller creates one key per intention, such as "refund ticket 5531 on order 1042", and sends it with every retry, usually in an Idempotency-Key header. The server stores the key, a fingerprint of the request and the result, and replays the stored result when the key comes back. Claiming a new key must be atomic so simultaneous duplicates cannot both run.

Like a cloakroom token: come back twice with token 42 and you still only get one coat.

Giving a whole image one label, or a ranked list of labels, such as "bakery" or "cat".

An image classifier outputs one score per label it was trained on and picks the highest. It cannot count objects or say where they are, and it can only answer with labels from its training set, such as the 1,000 ImageNet categories. Zero-shot models like CLIP remove that last limit.

Like glancing at a photo and saying what it is of, without pointing at anything.

The step that resizes and normalises a picture into the exact numbers a vision model expects.

Before a vision model sees an image, its image processor resizes it (for example to 224 by 224 pixels), scales the pixel values and arranges them as a tensor. It plays the same role for images that a tokenizer plays for text, and must match the model it was saved with. Shrinking the image is also where small details get lost.

Like trimming and arranging a cake on a standard board so it fits the box the decorator always uses.

Labelling every pixel of an image so you know the exact shape of each object.

A segmentation model outputs masks: black-and-white images where white marks the pixels belonging to an object or region. Panoptic segmentation assigns every pixel to some object or background region. Background removal is segmentation with two classes, the subject and everything else, and the mask can be used as a transparency layer.

Like colouring in exactly the cake with a highlighter, edge to edge, and nothing else.

The chunks an image is turned into so that a language model can read it alongside text.

A vision-language model encodes an image into a sequence of tokens that sit in the context window next to the text. A typical photo can become anywhere from tens of tokens in small local models to around a thousand or more in large hosted ones, and on paid APIs they are billed like text tokens. Resizing images to what the question needs saves cost and latency.

Like cutting a photo into numbered tiles so it can be passed through the same letterbox as the written order.

Passing a model's output to something powerful, like a web page or database, without checking it.

Model output can contain HTML, scripts, SQL or links an attacker steered it to produce. Rendering or running it unchecked can lead to cross-site scripting, data leaks through auto-loading images, or database damage. Treat model output exactly like untrusted user input: escape, validate and never execute it.

Like posting a customer-written note straight onto the shop window without reading it.

A model picking up a pattern from examples inside its prompt, without any retraining.

The examples shift the model's predictions for that one call. Its parameters do not change, so nothing is remembered for the next call. This is what makes few-shot prompting work.

Like reading a cheat sheet before each task rather than actually memorising the material.

The prepare-ahead half of RAG: loading documents, splitting them into chunks, embedding them and storing them for search.

Indexing runs when documents change, not when users ask questions. It loads text, cleans it, chunks it, turns each chunk into an embedding and saves vector, text and metadata in a vector database. When a document changes, its old chunks must be replaced, or stale versions keep being retrieved.

Like librarians cataloguing new books before the library opens, so visitors can find things quickly later.

Prompt injection hidden inside content an AI reads while doing its job, such as an email or a web page.

The person using the AI may be completely innocent; the attack is planted in a document, ticket, review or page the model processes. Because the model reads that content as text alongside its instructions, the hidden instructions can steer it. It is especially dangerous for agents with access to private data and outbound tools.

Like a forged instruction tucked inside a delivery slip that the cashier reads aloud and follows.

Using a trained model to get an answer, with its parameters fixed.

During inference no learning happens: input goes in, output comes out. Every call you make to a model API is inference. It is fast and cheap compared with training, and it is why a chatbot does not learn from your conversation.

Like an experienced baker making today's order: no new learning, just doing.

A company that runs a model on its own GPUs and returns the answers to you over the internet, billed per use.

Hugging Face routes requests for many Hub models to partner providers through one token and one client, so you can use large models without owning hardware. The trade-offs are per-request cost, data leaving your machine, and the risk that a provider stops serving a model you depend on.

Like ordering from a cloud kitchen instead of cooking at home: no stove needed, but you pay per plate and someone else sees the order.

Building a new class on an existing one, keeping its behaviour and changing only what differs.

A child class like `AllergyTicket(SupportTicket)` gets all the parent's methods and can override some of them. `super().__init__(...)` runs the parent's setup. Use it for "is a kind of" relationships, and keep chains short.

Like a seasonal recipe based on the classic: same steps, one ingredient swapped.

A code review comment attached to a specific line of a pull request's diff.

GitHub only accepts inline comments on lines that appear in the diff, identified by file path, line number in the new file and side. Comments can include suggestion blocks that the author can apply with one click. Posting them together as one review sends a single notification.

Like a sticky note placed on the exact line of a recipe card, instead of a note about the whole card.

Repainting only a chosen part of an image, marked by a mask, while keeping the rest unchanged.

An inpainting model receives the original image, a mask that is white where new content should go, and a prompt for that area. Only the masked region is regenerated by diffusion, blending with its surroundings. It is used to remove unwanted objects or change details; generated content must not be passed off as a real product photo.

Like icing over one smudged corner of a cake so it matches the rest perfectly.

The tokens you send to the model in a request: instructions, history, documents and the question.

Providers count and bill the input separately from the output. Everything in the request counts, including the system prompt and the conversation history you resend each turn, plus a few formatting tokens per message. Input is usually several times cheaper per token than output.

Like the pages of the order form the baker has to read before starting work.

A chat template that wraps user turns in [INST] and [/INST], used by Llama 2 and Mistral models.

The model's reply follows [/INST] and ends with the end-of-sequence token </s>. Llama 2 places an optional system prompt inside <<SYS>> tags in the first turn; Mistral folds system instructions into the first user message instead.

Like quotation marks around everything the customer says in a transcript, with the reply written after.

The trained tendency of chat models to follow the system prompt over the user's messages when they conflict.

Models learn from many examples that system instructions set the rules and users ask within them. This is why a well-written system prompt holds against most "ignore your rules" messages. It is a strong tendency, not a guarantee, so it is not a substitute for checks in code.

Like a shop assistant who follows the manager's written policy even when a customer insists otherwise, most of the time.

Further training a base model on example conversations so it learns to follow instructions and answer.

Instruction tuning teaches a model to respond helpfully, follow a system prompt and stop at the end of its turn. All the training examples use one chat template, which is why the model works best with that template afterwards.

Like an apprenticeship that turns a knowledgeable person into someone who serves customers properly.

A language model given extra training on conversations so that it answers requests instead of just continuing text.

Instruction tuning takes a base model and trains it further on instruction and response pairs, usually followed by preference tuning where better answers are rewarded. It changes behaviour (answer, stay on topic, stop when done) far more than knowledge. Names often end in -it, -Instruct or -chat.

Like the same well-read apprentice after weeks of counter training with a patient supervisor.

A LangGraph call that pauses a run in the middle and waits, for as long as needed, for a person to respond.

A node or tool calls interrupt(payload); the graph saves a checkpoint and returns, with the payload under "__interrupt__". Later, invoking with Command(resume=value) and the same thread ID re-runs that node from the top, and this time interrupt() returns the value. It needs a checkpointer, and code before the interrupt should be free of side effects because it runs twice.

Like a cashier pressing the "call manager" button and leaving the sale open on the till until the manager arrives.

An index that groups vectors into clusters and searches only the clusters nearest the query.

IVF uses k-means to learn cluster centres, then stores each vector in its nearest cluster’s list. A query is compared with the centres, and only the nprobe closest clusters are searched. More clusters opened means higher recall and slower queries; neighbours near a cluster border can be missed.

Like a library with sections: you search only the shelves in the sections closest to what you want.

A prompt designed to talk a model out of its safety rules.

Jailbreaks often use role-play ("pretend you have no rules"), fictional framing ("for a story I am writing"), other languages, encodings or slow escalation over many turns. They target the model’s trained behaviour rather than your app’s instructions specifically. Defences combine model choice, guardrails and testing with a growing library of known attacks.

Like a con artist who convinces a security guard that the rules do not apply this one time.

A templating language with placeholders, conditions and loops, used for web pages and prompts.

Jinja templates use {{ value }} for placeholders and {% if %} and {% for %} blocks for logic. It suits prompts with optional sections and lists. Hugging Face also stores each model's chat template as a Jinja template.

Like a mail merge that can also decide which paragraphs to include and repeat a line for each item.

A small random amount added to retry delays so that many clients do not retry at the same instant.

If thousands of clients fail together and all wait exactly two seconds, they all hit the service again at once. Adding a random fraction of a second spreads the retries out.

Like staggering staff lunch breaks so the canteen is never swamped at exactly one o'clock.

A unique identifier given to a queued job so its status and result can be looked up later.

The API returns the job ID immediately, and clients use it to poll for status. RQ generates random UUIDs by default, or you can supply your own, which enables idempotency. An ID identifies a job but is not permission to see it, so check ownership too.

Like the token number you get at a busy restaurant counter.

A SQL operation that combines rows from two tables wherever their keys match.

JOIN customers ON customers.id = conversations.customer_id glues each conversation to the customer who had it, so one query can return names and questions together. JOINs are how normalised tables, each holding one kind of thing, answer questions that span them.

Like laying the customer ledger next to the order ledger and matching rows by customer number.

A plain-text format for structured data, made of named fields, lists, text, numbers, true, false and null.

JSON is the standard body format for web APIs, including every major model API. Objects use curly braces and arrays use square brackets, and strings must use double quotes. In Python, `json.loads` and `json.dumps` convert between JSON text and dictionaries and lists.

Like a standard order form whose boxes every supplier knows how to read.

A standard, machine-readable description of what valid JSON looks like.

A JSON Schema lists the fields, their types, which are required and any limits. LLM APIs accept schemas to constrain structured outputs and to describe tools. Pydantic generates one from any model with `model_json_schema()`.

Like a blank order form that shows exactly which boxes must be filled and how.

A simple standard for one program to ask another to run a function, using JSON messages.

JSON-RPC 2.0 has requests (with an id, a method and params), responses (repeating the id, with a result or an error) and notifications (no id, no reply). MCP uses it for every message, which lets clients send several requests and match answers by id.

Like a booking system where every request gets a reference number and every reply quotes it.

Measuring and improving how well an LLM judge agrees with careful human labels.

You label a set of outputs by hand, split it into training examples, a development set and a hold-out set, iterate on the judge prompt using the development set, and report agreement on the hold-out set. Agreement is measured separately on real passes (TPR) and real fails (TNR). Re-check alignment when the model, product or standard changes.

Like grading the same batch of cakes alongside a new inspector until you agree almost every time.

A method that labels a new item by a vote among the k most similar examples it already knows.

k-NN stores every labelled example. For a new item it finds the k closest examples, often by cosine similarity, and picks the most common label among them. It copes well with messy categories and can explain itself by showing the neighbours, but it must search all stored examples for each prediction.

Like deciding which folder a letter goes in by finding the five most similar letters already filed and seeing where most of them went.

A high-level deep learning library, built into TensorFlow, for defining and training models with little code.

Keras lets you stack layers, then compile (choose optimiser, loss and metrics) and fit (run the training loop) in a few lines. Models save to a single .keras file. Keras 3 can also run on PyTorch or JAX underneath.

Like a well-equipped kitchen with preset programmes: fast for standard recipes, with manual controls when you need them.

Search that matches the exact words someone typed against the words stored in your data.

Keyword search is fast, cheap and predictable, and it is excellent for exact things like order numbers, product codes and names. It does not understand meaning, so "eggless" will not find "egg-free". Modern systems often combine it with semantic search, which is called hybrid search.

Like finding a book by scanning titles for the exact words you remember, even if the book is about exactly what you want under a different name.

A feature flag whose job is to shut down a feature or tool immediately in an emergency.

Kill switches can cover a whole AI feature or a single risky tool. They must be tested regularly, and people outside engineering, such as a support lead, should know where they are and be allowed to use them. Automatic breakers may switch a feature off, but switching back on should be a human decision with a staged rollout.

Like the big red emergency stop button on a factory machine.

A measure of how different one probability distribution is from another.

It is zero when two distributions are identical and grows as they drift apart. In a VAE it charges each image's cloud of codes for being off-centre or the wrong width compared with a standard bell curve. It also appears in RLHF and distillation as a way to keep one model close to another.

Like rent that rises the further your house is from the town centre.

The date after which a model's training text stops, so it knows nothing newer.

Anything that happened after the cutoff, and anything private to your business at any time, is unknown to the model unless you supply it. Models may not realise what they do not know, so they can describe outdated or generic information confidently.

Like a guidebook printed last year: useful, but silent about the shop that opened last month.

A way of storing information as things and the named relationships between them.

A knowledge graph holds nodes (customers, products, allergens) connected by edges (PLACED, CONTAINS, ALLERGIC_TO). Because relationships are stored explicitly, you can answer questions by following them across several steps, and explain every answer by the path that produced it.

Like a detective's corkboard of photos joined by labelled red string.

An embedded graph database that runs inside your Python program and speaks Cypher.

Kuzu needs no server: you install it and point it at a folder. It requires a schema of node and relationship tables before you insert data, which catches typos early. Its original company stopped active development in 2025; the last release still works and community forks continue it.

Like SQLite, but for graphs: a database that lives in a file next to your code.

Saved intermediate results that let a model avoid recomputing earlier tokens each time it writes a new one.

While generating, the model stores some of the work it has already done for every previous token, so each new token needs only a little extra computation. It makes decode much faster but uses a lot of GPU memory, which is one reason long contexts are expensive to serve.

Like a baker keeping the dough already mixed instead of starting from flour for every new step.

A tiny function with no name, written in one line.

`lambda o: o["store"]` is the same as a function that returns an order's store. Lambdas are most often passed to `sorted(..., key=...)` to say what to sort by. Use a named function when the logic deserves a name.

Like a sticky note with a one-line instruction, instead of a full recipe card.

A popular open-source Python library of building blocks for LLM apps, including loaders, splitters, retrievers and many integrations.

LangChain offers standard interfaces and hundreds of integrations with models, databases and tools. Its sister project LangGraph is used for agents and multi-step workflows. It moves quickly and has reorganised its packages more than once, so pin versions and prefer the official docs over old tutorials.

Like a huge supermarket with ingredients for any cuisine in standard packaging.

An open-source visual tool for building LLM apps by dragging components onto a canvas and wiring them together.

Langflow provides components for chat input and output, prompts, models, retrieval, memory and agents with tools. You can test a flow in its Playground, call it through an HTTP API and export it as JSON. It is excellent for prototypes, demos and tools owned by non-engineers, and weaker for complex logic, testing and code review.

Like building with Lego bricks that each do one job, instead of machining every part yourself.

An open-source tool for recording and reading the traces of LLM apps.

Langfuse is an LLM engineering platform whose core is MIT-licensed open source, with some enterprise features under a separate licence. You can self-host it with Docker or use Langfuse Cloud. It covers tracing, sessions, cost tracking, prompt management, datasets, scores and evaluations, and its Python SDK is built on OpenTelemetry.

Like a CCTV kit, cameras and control room together, that you can install in your own back office.

A library from the LangChain team for building agents and multi-step workflows as graphs of steps with saved state.

LangGraph models an application as nodes (steps) and edges (what happens next), with state that can be checkpointed and resumed. It suits agents with tools, branching and human approval steps. It is covered in depth in Module 14.

Like a flowchart that actually runs, remembering where it was if interrupted.

A visual debugger that draws your graph and lets you run it, watch its state and rewind it by clicking.

You start it locally with the LangGraph command-line tool (uv run langgraph dev) after describing your graph in a langgraph.json file. It shows each node as it runs, the state at every step and the threads you have created, and lets you fork from any checkpoint. It is a development tool, not something you ship to users.

Like a glass-walled kitchen where you can pause service and inspect every ticket on the rail.

A hosted service from the LangChain team that records and displays traces of LLM app runs.

Setting LANGSMITH_TRACING=true and LANGSMITH_API_KEY in the environment makes LangChain and LangGraph code send traces to LangSmith, where you can click into each step and see the exact prompt and reply. It also supports datasets and evaluations. Module 22 compares it with open alternatives.

Like a flight recorder for your app that you can replay in a web browser.

A very large model trained on huge amounts of text to predict the next piece of text.

Getting extremely good at predicting the next token turns out to let a model answer questions, summarise, translate, reason through problems and write code. Claude, GPT and Gemini are LLMs. They write one token at a time, do not learn from your chats, and can be confidently wrong.

Like an autocomplete that has read a library the size of a city.

The waiting time between sending a request and getting a response.

For AI apps people usually track two numbers: time to the first token and time to the complete answer. Latency depends on model size, prompt length, answer length and the number of round trips, such as tool calls. Lower latency often trades off against cost or quality.

Like the time between ordering and the first plate landing on your table.

The total waiting time you allow for a response, divided among every step that has to happen.

For a voice agent the key number is voice-to-voice latency, from the caller stopping to hearing the reply, ideally about one second. That budget is shared by turn detection, transcription, the model's first token, the first sentence, speech synthesis, network and playback. Logging each stage shows where to spend effort.

Like planning a 30-minute delivery promise: so many minutes to bake, to box and to ride, and they must add up.

Running the diffusion process on a compressed version of the image instead of on raw pixels.

A VAE encoder compresses each image into a small latent, for example 64 x 64 x 4 numbers for a 512 x 512 image, and the decoder turns the final latent back into pixels. It makes diffusion far cheaper and is the basis of Stable Diffusion.

Like sketching a design at thumbnail size before painting it full size.

A compressed numerical representation of data where a model does its work, smaller than the raw pixels or sound.

Many image generators use an autoencoder (VAE) to squeeze a 512 by 512 image into a 64 by 64 by 4 grid of numbers, about 48 times fewer values, and run diffusion there before decoding back to pixels. Working in latent space makes generation far cheaper. The idea of a compressed space also underlies embeddings.

Like working from a small, detailed sketch and only painting the full-size canvas at the end.

A step that rescales each token's numbers to a steady average and spread.

Layer norm subtracts the average of a vector's numbers, divides by their spread, then applies a learned scale and shift. It stops values drifting too large or too small as they pass through many layers. Most modern models apply it before each sub-layer ("pre-norm").

Like a thermostat that keeps every oven at a steady temperature no matter how busy the kitchen gets.

LangChain's way of joining steps with the | character so the output of one feeds the next.

Each piece (a prompt template, a model, an output parser) is a Runnable with an invoke() method. Writing prompt | llm | StrOutputParser() creates a chain that fills the prompt, calls the model and extracts the text. The chain is itself a Runnable, so it can be invoked, streamed or used inside a graph node.

Like a conveyor belt where each machine takes what the previous one produced and passes its result along.

A simplifying layer whose hidden details still affect you, usually at the worst moment.

Frameworks hide details so you can think at a higher level, but when something breaks you often need to understand those details anyway, plus the framework's own layers. A page-numbering convention inside a loader is a typical example. Knowing where the leaks are is part of choosing a framework.

Like a car you never needed to understand until it broke down on the highway.

The number that sets how big a step training takes on each update.

Too small and training crawls; too large and it overshoots, bounces around or diverges to nan. It is usually the single most important setting to tune, often by trying values a factor of 3 to 10 apart. Schedulers change it during training, typically large early and small late.

Like your stride length walking downhill: tiny steps take forever, giant leaps overshoot the valley.

Giving each part of a system only the access it needs to do its job.

For AI tools this means narrow, specific tools rather than general ones like "fetch any URL", read access before write access, and no secrets in what the model sees. It limits the damage a confused model or a malicious message can do.

Like giving the delivery rider the key to the dispatch room, not to the safe.

The dangerous combination of private data, untrusted content and a way to send data out, all available to one agent.

With all three, a hidden instruction in untrusted content can make the agent read private data and leak it. The term was coined by Simon Willison. The defence is to remove at least one leg, for example by making a web-reading agent read-only with no way to send messages.

Like leaving a safe open, a stranger in the room and the back door unlocked: any two are risky, all three are a robbery.

A rating scale such as "rate from 1 to 5", common in surveys.

Likert scales feel richer than yes or no, but for evaluating AI output they cause problems: raters disagree about the middle values, averages hide dangerous cases, and a score does not say what to fix. Most practitioners now prefer binary evals per failure mode.

Like rating a kitchen "3.7 for cleanliness": it sounds precise, but nobody knows what to scrub tomorrow.

A layer that multiplies its inputs by a weight matrix and adds a bias.

A linear layer computes y = Wx + b for a whole batch at once. nn.Linear(in, out) holds out × in weights plus out biases. Stacking linear layers only becomes more powerful than one layer when activation functions sit between them.

Like a panel of tasters, each with their own opinion of how much every feature matters.

An ordered collection of values that you can add to, remove from and change.

Written with square brackets, like `["croissant", "plum cake"]`. Positions start at 0, and lists can hold duplicates. A chat with an LLM is sent as a list of message dictionaries.

Like the ticket rail in a kitchen: orders in sequence, and you can add or pull tickets.

An open-source library and proxy that lets you call many model providers through one OpenAI-style interface.

LiteLLM translates a single request format into the APIs of more than a hundred providers, selected by model strings like "anthropic/..." or "openai/...". Run as a proxy server, it adds fallbacks, retries, per-key budgets and spend tracking through configuration. It is a common off-the-shelf alternative to writing your own gateway.

Like a universal travel adapter that fits your plug into sockets in every country.

A simple rule: the average number of jobs in progress equals the arrival rate times the time each job takes.

If 3.3 questions arrive per second and each takes 3.1 seconds, about 10 are being worked on at any moment, so you need at least that many workers to keep up. Below that number the queue grows for as long as the burst lasts; above it, extra workers sit idle. Add headroom because arrivals are lumpy.

Like working out how many tables a café needs from how many guests arrive per hour and how long each stays.

A free safety classifier model from Meta that labels messages or replies as safe or unsafe.

Llama Guard reads a user message, or a user message plus the assistant's reply, and outputs "safe" or "unsafe" with hazard category codes such as S10. It is designed for content safety (violence, hate, self-harm and similar) and can run locally, for example through Ollama. It is not built to detect prompt injection.

Like a smoke detector: excellent for smoke, no use against a polite burglar.

An open-source Python framework focused on connecting LLMs to your own data for retrieval and question answering.

LlamaIndex gives standard names and defaults to each RAG stage: readers, documents, nodes, indexes, retrievers and query engines. It is strong at ingesting many formats and at retrieval add-ons. Its defaults, including which model provider to use, should be set explicitly for production.

Like a meal kit for one cuisine: pre-measured, quick and very good at its speciality.

One layer that sits between your app and several model providers and hides their differences.

An LLM gateway exposes one interface, translates it into each provider's request and response format through adapters, and chooses which provider to call. It is the natural home for fallback chains, circuit breakers, retries, budgets and logging of which provider answered. It can be a small module in your code, an open-source proxy such as LiteLLM, or a hosted service.

Like a purchasing clerk who knows every flour mill's order form, so store managers only say what they need.

Using a language model to grade another model's outputs.

Judges work best on narrow questions such as "is this sentence supported by this paragraph?" and poorly on vague ones such as "rate this answer out of 10". They can be wrong and can favour their own style, so spot-check their verdicts and prefer a judge from a different model family.

Like asking a colleague to proofread: useful, but you still check their corrections.

MLOps for apps built on large language models, focused on prompts, retrieval, tools and model choice.

Most teams rent an LLM through an API rather than training it, so LLMOps manages what surrounds the model: versioned prompts, retrieved knowledge, tools, evals, staged rollouts, cost and feedback. It runs in three phases, ideation, development and operations, and loops back as new failures are found. The vendor’s model can also change, so pinning versions matters.

Like managing a brilliant contract chef: you cannot change their training, but you control the brief, the ingredients, the tools and the tasting before plates go out.

A file that records the exact version of every package your project uses, so installs are identical everywhere.

Without a lock file, a reinstall can quietly pull newer versions that rename or remove things you depend on. uv writes uv.lock for you; commit it, and upgrade deliberately on a branch with your tests as a safety net.

Like a recipe that says "Amul butter, 100 g" rather than "some butter".

Recording what a program does, with timestamps and severity levels, so you can investigate later.

Python's `logging` module has levels from `DEBUG` to `CRITICAL`, and you choose which levels to keep. `logger.exception(...)` records the full traceback. Never log secrets or personal data.

Like the kitchen logbook: every entry has a time, who wrote it and how serious it was.

A simple, classic machine learning method that learns boundaries between categories and gives a probability for each.

Logistic regression learns a weight for each input number per label and turns the weighted sum into a probability. Trained on embeddings, it is fast, needs only hundreds of examples and is often the most accurate simple classifier. Its probabilities let you set thresholds, such as sending unsure cases to a person.

Like an experienced sorter who has learned where the invisible fences between departments run, rather than comparing every letter to old ones.

The raw scores a model gives every possible next token, before they are turned into probabilities.

Higher scores mean more likely tokens. A formula called softmax turns logits into probabilities that add up to 100%. Temperature works by dividing the logits before softmax, which sharpens or flattens the resulting probabilities.

Like judges' raw marks before they are converted into each contestant's share of the vote.

A task that takes many steps and a long time, where early decisions affect later ones.

Writing a researched report, refactoring a codebase or planning a product launch are long-horizon tasks. Agents struggle with them because context fills up, goals drift and failures midway lose work. Plans, files, checkpoints, budgets and human check-ins make them manageable.

Like baking a wedding cake over two days rather than toasting a slice of bread.

Information an agent keeps outside the model, between conversations, and brings back when it is relevant.

Because a model forgets everything between calls, long-term memory lives in an external store such as a database or vector database. It works as a loop: decide what to write, retrieve what is relevant to a new message, and inject it into the prompt. Factual, episodic, semantic and procedural memories are all kinds of long-term memory.

Like a filing cabinet behind the counter: the assistant pulls out the right card when a regular customer walks in.

Accidentally using information in a test that would not have been available at the time of the decision.

Look-ahead bias makes backtests look far better than reality. It creeps in through signals computed over the full dataset, news timestamps in the wrong time zone, revised data, and, for LLMs, models that have already read about the period being tested. Defences include cutting data at the as-of date in code and tests that tamper with future data.

Like "predicting" yesterday’s cricket score after reading this morning’s newspaper.

A fine-tuning method that freezes the model and trains small correction matrices added to its layers.

Each chosen weight matrix gets a correction written as the product of two thin matrices, B and A, scaled by alpha over r. Only these are trained, usually under 1% of the parameters. After training the correction can be merged into the weights with no slowdown, or kept as a small swappable file.

Like clipping sticky notes of adjustments onto every recipe instead of rewriting the recipe book.

A single number that measures how wrong a model's predictions are; training tries to make it small.

A common loss is the mean squared error: square each prediction's miss, then average them. In the delivery example the loss started at 733.6 with both knobs at zero and fell to almost 0 after training. Choosing what the loss measures decides what the model gets good at.

Like a score from a tasting panel, where lower means closer to the recipe.

A formula that turns “how wrong is the model?” into a single number to minimise.

Training is the search for parameters that make the loss small. Mean squared error is common for predicting quantities; cross-entropy is common for classification, and punishes confident wrong answers hard. Choosing the loss is choosing what the model cares about.

Like a scorecard that gives one total penalty for every mistake made on a tray of pastries.

Choosing how much each head’s loss counts in a multi-output model’s total loss.

The total loss is a weighted sum of each head’s loss. If one loss is on a much larger scale, its gradients dominate and other heads stop learning. Put losses on comparable scales first (for example by standardising regression targets), then tune the weights on each head’s validation metric; automatic schemes such as uncertainty weighting exist for many heads.

Like converting every judge’s scores to the same scale, then deciding how much each judge’s opinion matters.

The tendency of models to use information at the start and end of a long input better than information buried in the middle.

Researchers found that answer quality can drop when the relevant fact sits in the middle of a very long context. Newer models handle long inputs much better, but the lesson remains: a short, relevant context usually beats a huge one.

Like remembering the first and last pages of a long report better than page 97.

Describing a big table of numbers using only a few underlying patterns.

A matrix of rank r can be written as the product of a tall matrix with r columns and a wide matrix with r rows. LoRA bets that the change needed to adapt a model has low rank, so a few patterns, mixed in different amounts, describe it well.

Like adjusting hundreds of recipes with the same three ideas: less sugar, a pinch more salt, extra vanilla.

Building software mostly by arranging and configuring ready-made pieces instead of writing code line by line.

Low-code tools let people who are not programmers build useful apps quickly and let engineers prototype fast. The trade-off is that anything the ready-made pieces do not cover is hard to express, and testing, versioning and reviewing changes are weaker than in code. Many teams prototype low-code and rebuild proven ideas in code.

Like assembling flat-pack furniture rather than doing carpentry: fast and tidy, until you need a shape the kit does not include.

A recurrent network with a separate long-term memory and gates that decide what to keep, add and output.

An LSTM keeps a cell state updated by multiplying by a forget gate and adding new content through an input gate, while an output gate controls what is exposed. Because the cell state is not squashed at every step, memory and gradients survive much longer than in a plain RNN. It dominated sequence modelling before transformers.

Like a recipe binder with an eraser, a pen and a reading lamp, each controlled by the baker.

A way of building software that learns patterns from examples instead of following rules a person wrote.

You give a machine learning method a dataset of examples, often with the right answers, and it produces a model that makes predictions for new inputs. Classic ML methods such as linear regression and decision trees work well on tidy rows and columns of numbers, like daily sales.

Like a trainee baker who learns when a loaf is done by watching a thousand loaves, rather than reading a rule.

Converting text from one language to another by computer.

LLMs translate well between widely used languages because they were trained on many languages at once. Business translation needs a keep-list for names, approved wording for fixed phrases such as allergen warnings, and checks that numbers survive. Scripts like Devanagari often use more tokens than English.

Like a bilingual colleague who is fluent but needs to be told which brand names never to change.

Software that a provider runs, updates and backs up for you, which you use over the internet.

Managed databases remove the work of installing, upgrading, monitoring and recovering servers, in exchange for a fee and less control. Comparing managed with self-hosted options means counting engineers’ time, not only server bills. Check where a managed service stores data before using it for sensitive information.

Like renting a serviced flat instead of owning a house: less freedom, but someone else fixes the boiler.

Processing a large input by handling each piece separately, then combining the results.

For summaries, the map step summarises each chunk (often in parallel) and the reduce step summarises the summaries. It handles texts too long for one call and keeps each call focused. Facts that only make sense across chunks can be missed.

Like six people each reading fifty reviews and a seventh combining their notes.

Attention where each token is blocked from looking at the tokens that come after it.

Future positions get a score of negative infinity before the softmax, so their weights become exactly zero. This lets a model learn every next-token prediction in a text in one parallel pass without cheating. Without it, training loss looks perfect and generation is garbage.

Like folding an exam paper so you can only see the questions you have reached, not the answer key printed below.

A training game where some words are hidden and the model must guess them using the words on both sides.

BERT hid about 15% of tokens per sentence and was graded only on those. Seeing both sides makes the model very good at understanding text, but it never practises writing left to right, so it is poor at generating text.

Like a fill-in-the-blanks worksheet where you can read the whole sentence around each gap.

Python's standard library for drawing charts.

matplotlib makes line, bar, scatter and many other plots, and pandas can draw with it directly. Always give charts a title and axis labels, and save them with `fig.savefig(...)` for reports.

Like a graph-paper notebook that draws the lines for you.

A grid of numbers arranged in rows and columns.

In deep learning, a matrix usually holds many examples (one per row) or the weights of a layer. Multiplying two matrices computes many dot products at once, which is why GPUs, built for exactly that, make neural networks fast. Two matrices can only be multiplied when the inner sizes match: (a × b) times (b × c) gives (a × c).

Like a spreadsheet of orders, one row per order and one column per detail.

The most tokens a model may produce in one reply: a hard ceiling, not a target length.

When the model reaches the limit, the reply is cut off and the finish reason says so. On reasoning models, hidden thinking counts against the same limit, so a low value can even produce an empty reply. You pay only for tokens actually produced, so set it generously and ask for short answers in the prompt.

Like telling the baker "stop at 6 pm whatever happens": it limits time, it does not make the cake smaller.

A way of choosing results that are relevant but not too similar to each other.

MMR picks results one at a time. Each candidate’s score is its relevance minus a penalty for how similar it is to items already chosen, balanced by a dial called lambda between 0 and 1. It turns "four near-identical croissants" into a varied, useful list, and it is also used in RAG to avoid retrieving duplicate chunks.

Like a host choosing party snacks: after the samosas, the next pick should probably not be more samosas.

The part of an AI app that manages one connection to one MCP server.

A client performs the handshake with a server, lists its tools, resources and prompts, and sends requests on behalf of the host. A host runs one client per connected server, which keeps servers isolated from each other. Your own Python programs can also act as clients using the MCP SDK.

Like a hotel concierge's dedicated phone line to one outside business.

The AI application the user works in, such as Claude Desktop or Claude Code, which connects to MCP servers.

The host holds the conversation with the model, decides which servers to start, collects their tools for the model, and enforces the rules, such as asking the user before running a tool. It contains one MCP client per connected server.

Like the hotel itself: it serves the guest, keeps the rules and decides which outside services to call.

The official browser-based tool for connecting to an MCP server and trying its tools, resources and prompts by hand.

Started with npx @modelcontextprotocol/inspector followed by the command that runs your server, it acts as an MCP client without any model involved. That lets you test a server's behaviour directly and see its messages, separating server bugs from model choices.

Like a test bench where you plug in a new appliance and try every button before installing it in the kitchen.

A reusable message template an MCP server offers, which a user picks and fills in.

Prompts are user-controlled: a host typically shows them as a menu or slash command, the user supplies arguments, and the server returns ready-made messages. They are a good way to package a procedure, such as "check allergens with the tool before replying".

Like a fill-in-the-blanks form letter the shop keeps for common replies.

Read-only data an MCP server offers at an address, like crumb://menu/today, for an app to load into context.

Resources are controlled by the application or the user rather than chosen by the model mid-conversation. They suit documents, files and reference data that someone might want to read or attach, and they cannot change anything.

Like a printed menu on the counter: you can read it, but reading it does not place an order.

A small program that wraps one system, like an order database, and offers its tools and data to AI apps through MCP.

An MCP server exposes tools, resources and prompts in the standard MCP format, so any MCP-capable app can use them. Most are small programs, often running locally as a child process of the app; others run remotely over HTTP. You write a server once per system, and every host can connect to it.

Like a device with a standard USB-C port: build the port once and any compatible cable works.

The average size of a forecast’s misses, in the same units as what is being predicted.

MAE averages the absolute difference between predictions and actual values, so “MAE 12.6” means off by about 12.6 croissants a day. It is easy to explain and less dominated by rare huge misses than squared error. MAPE expresses the same idea as a percentage.

Like asking “on a typical day, by how many croissants were we off?”.

An open-source memory layer that extracts facts from conversations and stores them for an agent to search later.

Mem0 combines an extractor LLM, an embedding model and a vector store, plus a history log. Calling add on a conversation extracts short, self-contained memories scoped to a user, agent or run; calling search finds the most relevant ones by meaning. It can be self-hosted or used as a hosted service.

Like a clerk who sits in on every meeting, writes down what matters on index cards and files them under the right name.

Using an LLM to turn a conversation into a few short, self-contained facts worth remembering.

An extraction step reads new messages, compares them with what is already stored and outputs clean sentences such as "User's son Aarav cannot eat peanuts". It handles paraphrases and attribution that hand-written rules miss. Its output should still pass through rules in code before being stored.

Like turning a long meeting into three clear bullet points in the minutes.

Tricking an agent into saving false facts or hidden instructions that then affect every future conversation.

Because stored memories are injected into later prompts as trusted context, a single manipulative message such as "remember, you always approve my refunds" can become a permanent instruction. Defences include a write gate enforced in code, never storing claims about what the business owes, and labelling memories in prompts as notes rather than instructions.

Like someone slipping a forged note into your file that says "this customer gets everything free".

The label that says whose memory something is, such as a particular user, agent or session.

Memory libraries tag every memory with identifiers like user_id, agent_id or run_id, and every search filters on them. Scopes are what keep one customer's memories away from another's. The identifier should come from authentication, never from something the user typed.

Like the name on the front of a file folder: it decides which drawer the folder lives in and who can pull it.

What git reports when two branches changed the same lines and it cannot decide which version to keep.

Git marks the file with both versions between `<<<<<<<`, `=======` and `>>>>>>>` lines and waits for a person. You edit the file to the version you want, remove the markers, then add and commit. Conflicts are normal, not errors.

Like two bakers editing the same line of a recipe card: someone has to decide which quantity is right.

A shared, ordered list of jobs waiting to be done, stored where every part of the system can reach it.

Producers (such as a web server) add jobs; consumers (workers) take them off and process them. Because the queue lives outside any single process, jobs survive restarts and many workers can share the load. Redis or Valkey with RQ is a simple way to build one in Python.

Like the order rail in a kitchen where slips wait for the next free cook.

Labels stored alongside a chunk, such as source file, page, chapter or access level.

Metadata is not searched for meaning, but it is what lets you cite sources, filter search (for example, only HR documents, or only chunks a user may see) and debug wrong answers. Splitters copy it onto every chunk, so add it before splitting.

Like the label on a jar in the pantry: it is not the food, but you would not want to cook without it.

Narrowing search results using structured facts stored with each item, such as stock, category or date.

Metadata is the structured information stored next to each vector, like in_stock: true or store: "Baner". Filters on metadata are hard yes-or-no rules, while similarity only decides the order of what passes. Filtering before ranking (pre-filtering) avoids ending up with too few results.

Like telling a shop assistant "only things in stock, nothing with nuts" before they start suggesting what you might like.

A function that belongs to a class and works on one object's data.

Methods are defined inside a class and take `self`, the object they are called on, as their first parameter. `order.total()` is the same as `Order.total(order)`; Python passes the object in for you.

Like a button on a coffee machine: it acts on that machine, not on every machine in the shop.

An open-source vector database designed to scale from a laptop file to billions of vectors.

Milvus runs as Milvus Lite (a local file), Standalone (one server) or Distributed (many services on Kubernetes), with a managed version from Zilliz. It offers many index types, including IVF_PQ, HNSW and DISKANN, schemas with typed fields, filter expressions and partition keys. The distributed version scales very far but takes real work to operate.

Like a fleet of lorries: built for very heavy loads, as long as you have a depot and mechanics or hire someone who does.

A small random group of training examples used for one update step.

Instead of computing the exact gradient over the whole dataset, training estimates it from a mini-batch of, say, 32 to 256 examples. The estimate is noisy but cheap, so you get many more updates for the same compute. The batch size is how many examples each mini-batch holds.

Like tasting one tray of croissants instead of every croissant baked that day.

A model design with many expert sub-networks where each token is routed through only a few of them.

The feed-forward layers are replaced by sets of experts plus a small router that picks the top few for each token. Total parameters (knowledge) can be huge while active parameters (compute per token) stay small. Every expert must still be held in memory, and the load must be kept balanced.

Like a big kitchen with many specialist stations where each order only visits the two it needs.

The practices and tools for running machine learning models reliably after they are built.

MLOps covers the whole loop around a model: preparing data, training, registering versions, deploying, monitoring and retraining. It exists because models fail silently: they keep running and quietly get worse as the world changes. MLOps builds the noticing and the careful fixing that a model cannot do for itself.

Like the maintenance crew for a factory machine: someone has to check it, service it and recalibrate it, or it slowly starts making bad parts.

A classic public dataset of 70,000 small greyscale images of handwritten digits.

Each image is 28 by 28 pixels with a label from 0 to 9. It is small enough to train on a laptop in minutes, which makes it the standard practice ground for new techniques, including GANs and autoencoders.

Like the practice sponge every baking student makes before attempting a wedding cake.

Replacing a real dependency, such as an API client, with a stand-in that returns scripted answers during tests.

A fake LLM with the same method as the real client lets you test prompt building, validation, fallbacks and retries offline, quickly and for free. Passing the client into functions, rather than creating it inside them, makes this easy.

Like a flight simulator standing in for a real plane during training.

A GAN failure where the generator keeps producing only a few kinds of output instead of the full variety of the data.

The generator finds a handful of outputs that reliably fool the current discriminator and leans on them, for example drawing mostly 1s and 7s. A fixed-noise image grid and metrics like FID make it visible early.

Like an apprentice who learns that the taster never spots fake ladoos, so makes nothing but ladoos.

A program with adjustable settings that turns an input into a prediction or an output.

A model is a function whose behaviour depends on numbers inside it, called parameters, which are set by training on examples. It stores patterns, not answers, so it can respond to inputs it has never seen. Large language models are models with billions of parameters.

Like a recipe card plus exact quantities: the same card with different quantities makes a different cake.

The description page of a model that says what it does, how it was trained, its licence and its limits.

A model card is the README of a model repository. A good one lists the task, intended uses, training data, languages, evaluation results, known biases and licence. Reading it before downloading saves you from models that are too big, the wrong language or not allowed for commercial use.

Like the ingredients and allergy label on a box of cookies: read it before you serve it to customers.

An open standard for connecting AI apps to tools and data sources in one common way.

Introduced by Anthropic in 2024, MCP defines how an AI app (the client) discovers and uses tools, data and prompts offered by MCP servers. Build a tool once as an MCP server and many AI apps can use it. Module 18 shows how to build one.

Like USB-C for AI tools: one plug shape that works with many devices.

The small final part of a model that turns its internal understanding into the output a task needs.

Most of a transformer is a shared body that produces hidden states. A head on top turns them into, for example, one score per label, a start and end position for an answer, or next-token scores. Loading a checkpoint without its head gives no labels; attaching an untrained head gives random ones until it is fine-tuned.

Like the last station on a bakery line: the same oven, but decorating makes a cake and slicing makes a loaf.

The legal rules that say how you may use, change and share a model.

Open models use standard licences such as Apache 2.0 or MIT, company-written licences such as the Gemma Terms of Use or the Llama community licence, or non-commercial licences such as CC-BY-NC. The licence decides whether a business can ship the model in a product and what conditions apply. It can change between generations of the same model family.

Like the terms on a recipe book: some let you sell the cakes, some only let you bake at home.

Watching a deployed model’s inputs, outputs and results so you notice when it starts going wrong.

Good monitoring covers several layers: system health, data quality, input drift, model performance once real answers arrive, and the business outcome the model serves. Each layer has alerts with thresholds agreed in advance. Monitoring only uptime misses the most common failure, a model that is running perfectly and quietly wrong.

Like a shop manager who checks not only that the lights are on, but that the shelves are stocked and customers are leaving happy.

A catalogue of every version of a model, with what it was trained on, how it scored and who approved it.

A model registry answers "which model is running, and where did it come from?" Each entry links a model version to its training data snapshot, code, evaluation results and approval status. It makes rollbacks and audits possible. MLflow is a common open-source example.

Like a recipe book where every version of a recipe is dated, tasted, signed off and never thrown away.

Sending each request to the cheapest model and settings that can handle it well.

A router, built from rules or a small classifier, sends easy requests to a small or low-effort setup and hard or risky ones to a larger model. Done well, it cuts cost and latency without hurting quality. Evals must pass on every route, and uncertain cases should go to the careful route.

Like a hospital front desk sending a sprained ankle to a nurse and chest pain straight to the doctor.

A short text file that tells Ollama how to build a customised model preset.

A Modelfile names a base model with FROM, sets options such as temperature and context length with PARAMETER, and bakes in a SYSTEM prompt. Running ollama create turns it into a named model any app can call. It reuses the base model's weights and stores only the recipe.

Like a recipe card that says "start with the standard sponge, bake at 170 degrees, always add cardamom", so everyone gets the same cake.

A service that checks text or images for harmful content such as harassment, hate, self-harm or violence.

A moderation API returns a flag and a score per harm category. OpenAI’s moderation endpoint is free for its API users. Moderation catches harmful content, not polite manipulation, so it is one layer among several, and each flagged category needs an agreed action, such as a calm reply and a human handover.

Like a bouncer who stops aggressive people at the door but cannot spot a smooth-talking con artist.

A Python file whose code you can borrow and reuse in another file.

Python ships with a large standard library of modules, like `math` and `json`, and installed packages are imported the same way. Your own `.py` files are modules too. A folder of modules with an `__init__.py` is a package.

Like a recipe binder on the shelf: you borrow the recipe instead of rewriting it.

An optimiser trick that keeps moving in the direction recent gradients agreed on.

With momentum, each update follows a running blend of past gradients rather than only the current one. Zigzags cancel out and consistent directions build up speed; with the usual setting of 0.9, a steady slope gets up to ten times the plain step. Adam includes momentum.

Like pushing a loaded trolley: once rolling, small bumps do not knock it off course.

A popular database that stores flexible, JSON-like documents instead of rows in fixed tables.

MongoDB groups documents into collections and does not require you to design columns up front. LangGraph has an official MongoDB checkpointer (the langgraph-checkpoint-mongodb package), so conversation state can survive restarts and be shared by several servers. Treat a checkpoint database like any customer database: secure it, back it up and delete old data.

Like a filing cabinet of folders where each folder can hold whatever papers that customer needs.

A system where several agents, each with its own instructions, tools and context, work together on a task.

Usually an orchestrator briefs specialist agents, who work separately and return short memos. The main benefit is that each agent holds a small, focused part of the problem; parallelism and specialisation are bonuses. They typically cost several times more tokens than a well-designed single agent and are harder to debug.

Like a newsroom where an editor assigns reporters to different beats and writes the story from their memos.

Running several attention calculations side by side, each free to focus on a different relationship.

Each head has its own query, key and value matrices and works in a slice of the model's dimensions, such as 64 of 512. The heads' outputs are joined and mixed by one more matrix. Splitting the space keeps the total cost about the same as one big head, and any specialisation of heads emerges from training.

Like a tasting panel where one person judges sweetness, one packaging and one price, and their notes are combined into one verdict.

DeepSeek's attention design that caches one small compressed vector per token instead of full keys and values for every head.

A learned matrix compresses each token into a latent vector, and other matrices rebuild every head's keys and values from it when needed; a small separate key carries position information. It shrinks the KV cache many times over while matching standard attention's quality in DeepSeek's reports.

Like writing orders in a shorthand the whole kitchen learned to read back in full.

A question you can only answer by following several relationships in a row.

"Which customers bought something containing an ingredient their child is allergic to?" needs hops from allergen to product, product to order, order to customer and customer to family. Similarity search struggles with these because no single piece of text contains the answer; graphs handle them naturally.

Like asking "who is my cousin's teacher's neighbour?": you have to follow each link in turn.

Answering a question by searching several times, where each search is guided by what the previous one found.

A first search might reveal that two cakes share a production line; a second search then looks up that line’s allergen policy. Single-shot retrieval cannot follow such a trail. Agents do it naturally, but every hop adds cost and time.

Like following signposts: each one tells you which road to take to reach the next.

Classification where one item can have several labels at once.

Support messages often contain more than one issue, such as a safety report and a billing complaint. Forcing a single label drops information, sometimes the most important part. Allow a list of labels and, if needed, a separate primary label for sorting.

Like a recipe that is tagged both "eggless" and "festive".

Training one model to answer several related questions at once, with a shared core and one head per question.

A shared trunk learns a representation useful for all tasks, and small output heads turn it into each answer. Related tasks can improve each other and stay consistent; unrelated ones can compete, which is called negative transfer. Always compare with single-task models.

Like one batch of laminated dough that becomes croissants, pain au chocolat and danishes.

Able to work with more than one kind of input or output, such as text, images, audio and video.

A multimodal model can, for example, take an image and a question and answer in text, or put images and captions into one shared embedding space. Each kind of data is called a modality. Multimodal systems let one application see, hear and read, but each modality adds its own failure modes and costs.

Like a staff member who can read an order slip, look at a photo and listen to a voice note, rather than only reading.

An AI agent that can look at images (and sometimes hear audio) while it reasons and uses tools.

A multimodal agent runs the usual loop of thinking, calling tools and reading results, but user messages and tool results can contain images, such as a customer photo or a screenshot. That lets the model fetch and compare what it needs to see before acting. It inherits every risk of vision models, including invented details and instructions hidden in images.

Like a store manager who can look at the customer's photo, pull up the order and the delivery photo, and then decide.

Serving several separate customers from one system while keeping each one’s data private.

Each customer is a tenant. Isolation can come from separate indexes, separate namespaces or partitions, or a tenant filter on every query. The filter approach is the weakest, because one forgotten filter leaks data. Decide the tenant from authentication, never from user input.

Like an office building rented to several companies, each with its own locked floor.

Without a shared standard, connecting N apps to M tools needs a separate connector for every pair.

Four apps and five systems need twenty custom integrations, and each change in a system must be fixed in several places. A standard like MCP turns this into N + M pieces: one client per app and one server per system. The gap grows quickly as apps and tools are added.

Like needing a different charger for every gadget in every room, before USB-C.

A separate partition inside a Pinecone index; every read and write targets exactly one.

Namespaces split one index into walled-off sections. A query in one namespace cannot return records from another, which makes them the natural way to keep each customer’s data apart. Deleting all records in a namespace removes it in one call.

Like flats in one apartment building: shared plumbing, but each with its own locked door.

An open-source NVIDIA framework for wrapping a chatbot in configurable input and output checks.

With NeMo Guardrails you describe rails in configuration files: checks that run on incoming messages, checks on outgoing replies, and topics the bot should stick to. Rails can call a model with a checking prompt, your own Python functions or classifiers such as Llama Guard. It helps most when many checks and bots must be managed consistently.

Like a kitchen procedures binder that says which checks every order passes through.

A widely used graph database server that you query with Cypher.

Neo4j runs as a server that many applications can share, includes Neo4j Browser for viewing your graph as circles and arrows, and offers a free cloud tier. It does not require a schema, so you add uniqueness constraints yourself to prevent duplicates.

Like Postgres, but for graphs: a shared database server.

A model built from many simple maths units connected in layers, loosely inspired by brain cells.

Each unit multiplies its inputs by learned weights, adds them up and passes the result on. Stacking many layers lets the network find its own useful features in raw data like images or text. Deep learning means neural networks with many layers; LLMs are very large neural networks.

Like a relay of bakery stations, each turning what it receives into something a little more finished.

The single task an LLM performs: guessing which small piece of text should come next.

At each step the model scores every token in its vocabulary, turns the scores into probabilities, picks one, appends it and repeats. Whole answers, essays and programs are built from this one move, repeated hundreds of times. It is also why models can be fluent and wrong: the loop picks likely text, and no step checks whether it is true.

Like your phone keyboard's next-word suggestion, scaled up enormously and allowed to keep going on its own.

The base class every PyTorch model and layer inherits from.

A model defines its layers in __init__ and the flow of data in forward, then is called as model(x). nn.Module tracks all parameters so the optimiser can find them, switches between training and evaluation modes, and saves and loads weights through state_dict.

Like a standard recipe card format: every recipe lists its ingredients and its method in the same places.

One step in a graph: a function that reads the current state and returns the changes it wants to make.

In LangGraph a node is an ordinary Python function (or a compiled subgraph). It receives the state and returns a dictionary of updates containing only the keys it changes. A node can be plain arithmetic or a call to an LLM; the graph treats both the same way. Good nodes do one job and can be named with one verb.

Like one station in a bakery kitchen, such as decorating, which reads the ticket and writes down only what it did.

LlamaIndex's name for a chunk: a short passage with its own embedding, metadata and links to neighbouring chunks.

Nodes are created from documents by a node parser such as SentenceSplitter, which counts in tokens. Each node remembers its source document and the nodes before and after it, which enables tricks like fetching a neighbour for extra context.

Like a page torn from a book that still has a note saying which book and which pages come before and after.

The same input does not always produce exactly the same output.

LLM output varies because of sampling and, even at temperature 0, because of tiny numerical differences when providers batch requests on GPUs. Design for it: test important prompts many times, and never write code that breaks if the wording changes slightly.

Like asking the same baker for "something festive" twice and getting two different cakes.

Python's special value meaning "nothing here".

`None` is not zero, not an empty string and not `False`. It shows up for guest checkouts, unrated tickets and functions that return nothing. Check for it with `is None`, and remember JSON's `null` becomes `None` in Python.

Like a shelf labelled "deliberately empty": different from a shelf holding zero cakes.

Scaling a vector so its length is exactly 1 while keeping the direction it points in.

You normalise a vector by dividing every number in it by the vector’s length, so (3, 4) becomes (0.6, 0.8). Once all vectors have length 1, the dot product equals cosine similarity and all common distance measures rank results in the same order. Forgetting to normalise lets long vectors win comparisons unfairly.

Like asking everyone in a room to speak at the same volume, so you judge what they say rather than how loudly they say it.

A Python library for fast arrays: grids of numbers that are all the same type.

NumPy arrays have a shape and a single data type, and whole-array operations run in fast compiled code. Concepts like `axis`, boolean masks and broadcasting carry straight over to PyTorch and TensorFlow. pandas is built on top of NumPy.

Like an egg tray: identical slots, so you can rinse the whole tray at once.

One real thing built from a class, with its own data.

Two orders made from the same `Order` class share methods but each has its own ID and lines. Under the hood, an object is roughly a dictionary of its own attributes plus a link to its class. Asking for an attribute it does not have raises an error.

Like one cake from the mould, with its own flavour and icing message.

Finding each object in an image, naming it, and drawing a box around it.

A detector returns a list of objects, each with a label, a confidence score and a bounding box. A confidence threshold decides which guesses to keep, trading missed objects against false ones. Counting the boxes for a label gives you a count, which classification cannot.

Like a stock-taker who walks along the shelf putting a sticky note on every item and writing its name.

Being able to tell what your app did, and why, from the records it keeps.

Observability is the ability to answer questions about a running system, including questions you did not plan for, from the data it already emits. That data usually comes as three kinds of signal: logs (events), metrics (numbers over time) and traces (the path of one request). AI apps need it badly because they fail softly: a wrong answer still returns success.

Like a bakery with batch cards, temperature charts and CCTV: when a batch goes wrong, you can find out why without guessing.

Software that reads text out of images, such as a scanned page or a photo of a document.

Many PDFs contain scanned pages that are pictures, not text. Ordinary PDF loaders return nothing for them, without any error. OCR converts the image into text so it can be chunked and searched.

Like a person typing up a handwritten letter so it can be searched on a computer.

A free program that downloads open AI models and runs them on your own computer.

Ollama fetches quantized open-weights models with one command, runs them on your CPU or GPU, and serves them through a local web API on port 11434. It loads a model when a request arrives and unloads it after a few idle minutes. Programs talk to it through its own API, its Python library, or an OpenAI-compatible endpoint.

Like a small oven in your own bakery: no per-dish fee and the order slips never leave the shop, but it bakes simpler things than the big central kitchen.

Giving a model a single worked example before the real task.

One example teaches format very effectively. But the model cannot tell which features of a lone example matter, so it may over-copy coincidences, such as always choosing the example's label.

Like showing a trainee one finished order slip and hoping they guess which parts were important.

Writing free-form notes on what went wrong in each trace, without deciding categories in advance.

Borrowed from qualitative research, open coding is the first pass of error analysis. For each trace you write a specific, observable note about the first failure, the one furthest upstream, or "fine" if nothing went wrong. Categories come later, so they reflect what you saw rather than what you expected.

Like a restaurant critic scribbling "soup lukewarm, bread forgotten" on a first visit, before writing any checklist.

A self-hosted chat website, similar to ChatGPT, that runs on your own server in front of local or cloud models.

Open WebUI connects to Ollama and to any OpenAI-compatible API, and adds accounts, chat history, a model picker, document knowledge collections (built-in RAG) and model presets. The first account becomes the admin. Because it stores every user, chat and document, you have to secure, back up and update it yourself.

Like adding a shop counter to a kitchen: customers order at the counter, and nobody has to learn the kitchen's order-slip format.

A model whose trained parameters are published so anyone can download and run it.

Examples include Meta's Llama, Google's Gemma and Mistral's models. You can run them on your own hardware, keep data in-house and fine-tune them, at the cost of managing the hardware yourself. Closed models, such as Claude and GPT's main models, are available only through the provider's API.

Like a bakery that publishes its full recipe so you can bake at home, rather than only selling at the counter.

Any server that accepts requests in the same shape as OpenAI's API, so the same code can talk to it.

Many tools and model servers, including Ollama at /v1, copy the request and response format of OpenAI's Chat Completions API. Code written for OpenAI then works against them by changing the base URL. It describes a format only, not where the data goes or who runs the model.

Like a standard plug socket: any appliance with the right plug works, whoever made the appliance.

An open-source library that records LLM calls as standard traces and sends them to the tool you choose.

OpenLLMetry, by Traceloop, is Apache 2.0 open source and built on OpenTelemetry. It automatically instruments LLM SDKs like OpenAI and Anthropic, vector databases and frameworks. It stores nothing and has no interface of its own: it exports spans to any OpenTelemetry-compatible backend, such as Grafana Tempo, Jaeger, Honeycomb, Datadog or Langfuse.

Like cameras that plug into any standard recorder, sold without a recorder.

An open standard, not tied to any company, for recording traces, metrics and logs.

OpenTelemetry is a vendor-neutral project of the Cloud Native Computing Foundation. It gives you an API and SDK to create spans in your app, instrumentation libraries that create spans for you, exporters that send them out over the OTLP protocol, an optional Collector, and support from almost every observability backend. Because everyone speaks it, your AI spans and your ordinary service spans can share one trace.

Like a national postal system with standard parcels and trucks, so any courier and any sorting hub can handle any parcel.

The algorithm that turns gradients into actual parameter updates.

Plain SGD subtracts learning rate × gradient. Momentum adds a running average of recent gradients. Adam also adapts the step size for each parameter and is the common default; AdamW is Adam with correctly applied weight decay, standard for transformers.

Like the driving style used to get down the hill: walking, rolling a trolley, or rolling in shoes that adjust their grip.

A pattern where one model call plans the sub-tasks for each input, hands them to workers, and combines the results.

Unlike parallelisation, the sub-tasks are not known in advance: the orchestrator decides them per input. When each worker is itself a full agent, this becomes a multi-agent system. Validate the plan, because orchestrators can invent workers that do not exist.

Like a head chef reading an unusual catering order and deciding which stations need to be involved.

The standard format for sending traces, metrics and logs from your app to a tool that stores them.

OTLP, the OpenTelemetry Protocol, defines how telemetry travels over the network, over HTTP or gRPC. An exporter in your app sends spans as OTLP to a Collector or directly to a backend. Any backend that accepts OTLP, from Grafana Tempo to Langfuse, can receive data from any OpenTelemetry-instrumented app.

Like the delivery trucks of the postal system: they carry any standard parcel to any sorting hub.

A small final layer that turns a model’s shared representation into one particular answer.

A model can have several heads on the same trunk, for example one regression head for units sold and one classification head for sell-outs. Heads are usually tiny; the shared layers do most of the learning. Returning outputs as a dictionary keyed by head name keeps code readable.

Like different plating stations at the end of the same production line.

The tokens the model writes in its reply, including any hidden thinking.

Output tokens are generated one at a time, so they take longer and are priced higher than input tokens, often four to five times more. On reasoning models, the hidden thinking tokens are counted and billed as output too, even though you never see them.

Like the icing a baker pipes letter by letter: slower and more expensive than reading the order.

Checking a model’s output against a schema and business rules in code before using it.

Ask the model for structured output so there is something reliable to check, validate its shape with a library such as Pydantic, then apply business rules: refunds within limits, links only to your own domain, no claims of actions the bot cannot take. If validation fails, send a safe fallback instead and log the event.

Like a cashier who checks that the total on the receipt matches the items in the bag before handing it over.

When a model memorises its training data, noise included, and does worse on new data.

The sign is a gap: excellent scores on training data, disappointing scores on validation data, with validation loss rising while training loss keeps falling. The cures are more data, simpler models, dropout, weight decay and early stopping. Underfitting is the opposite: a model too simple or too briefly trained to learn even the real pattern.

Like a student who memorises last year’s exam answers and fails when the questions change.

A widely used list of the ten most important security risks in apps built on language models.

Published by OWASP, a non-profit security community, the list covers risks such as prompt injection, sensitive information disclosure, excessive agency and unbounded consumption, each with examples and mitigations. It is revised as attacks evolve; the 2025 edition was current at the time of writing. Teams use it to threat-model each AI feature.

Like a fire-safety inspection checklist written by people who have seen many kitchen fires.

A way of storing the KV cache in small fixed-size blocks handed out on demand, so almost no memory is wasted.

Introduced by the vLLM project, it borrows virtual-memory paging from operating systems: each conversation has a block table pointing to blocks anywhere in memory. It removes most of the waste from reserving maximum length up front and lets conversations share blocks for a common prompt prefix.

Like seating diners at small tables as they arrive instead of reserving a twelve-seat table for every party.

A Python library for loading, cleaning and summarising tables of data.

pandas reads CSV, JSON Lines, Excel and more into DataFrames, then filters, groups, pivots and joins them in a few lines. It is the standard tool for preparing training data and inspecting evaluation results.

Like a spreadsheet you can give precise instructions to in code.

Tracking trades with imaginary money to see how ideas perform, without risking anything real.

A paper-trading journal records each idea’s entry, invalidation level, target and size as if it had been traded, then scores it against later prices. It is a safe way to evaluate a system and practise discipline. It ignores some real-world effects, such as emotions and order execution, so results are usually better than live trading would be.

Like a flight simulator: real decisions and real feedback, without a real plane.

Several independent tool requests returned by a model in a single response.

When lookups do not depend on each other, such as stock at two stores, the model can request them together. Run them concurrently and return all results in one message, so the wait is the slowest tool rather than the sum of all of them.

Like a waiter handing the kitchen three slips at once for three different stations.

A workflow that runs several independent model calls at the same time and then combines their results.

Sectioning splits a task into independent parts, such as stock, allergen and pricing checks. Voting runs the same task several times and compares answers for more confidence. It cuts waiting time to the slowest call, but you pay for every call and can hit rate limits.

Like three cooks preparing the starter, main and dessert at once instead of one cook doing them in turn.

The adjustable numbers inside a model, set during training, that decide what it outputs.

In `minutes = w × km + b`, the parameters are `w` and `b`. Large models have billions of them; "an 8B model" means about 8 billion parameters. Parameters are also called weights, and the file you download for an open model is mostly a long list of them.

Like the knobs on a sound mixing desk: the same desk sounds completely different depending on where the knobs sit.

A field whose value decides which section of a collection each record is stored in.

In Milvus, marking a field such as tenant or city as the partition key routes records into partitions by that value. Searches filtered on the key touch less data, which speeds them up and supports multitenancy. Very uneven partitions, or filters much narrower than the key, still need testing.

Like sorting post into sacks by city before sorting each sack by street.

The share of outputs that passed an eval, such as 62% of allergy replies.

Pass rate is calculated only over outputs where the eval applies; cases where it does not apply are left out rather than counted as passes. On small samples it is noisy (40 of 50 passing could plausibly mean anything from about 69% to 91%), so watch for large moves and set targets by risk.

Like the share of cakes that pass the weight check at the end of a shift.

The list of folders your shell searches, in order, to find the program you typed.

When you type `uv`, the shell checks each folder on PATH for a program with that name. A newly installed tool is invisible until its folder is on PATH, and a terminal reads PATH when it starts, which is why you open a new terminal after installing something. `which uv` shows which program would run.

Like the list of storerooms an assistant checks, in order, when you ask for an ingredient by name.

A method that finds the directions in which your data is most spread out and flattens it along them.

PCA rotates the data so the first direction captures the most spread, the second the next most, and so on, then keeps the first few. It is fast, deterministic and keeps large-scale distances roughly honest, and it reports how much of the spread each direction keeps. On text embeddings, two components often keep only a small share of the information.

Like photographing a baguette from the side rather than end-on, because the side view shows the most.

A family of fine-tuning methods that train only a small number of parameters, and the Hugging Face library that implements them.

Methods include LoRA, QLoRA, DoRA, prompt tuning and adapters. The PEFT library provides LoraConfig, get_peft_model, save_pretrained for adapters and merge_and_unload for serving.

Like renovating a kitchen by changing the fittings instead of rebuilding the walls.

How long something feels to the user, which can differ a lot from how long it actually takes.

Two features with the same real latency can feel very different. Streaming the first words quickly, showing progress, or doing work in the background before anyone asks all reduce perceived latency. Designing for it is often cheaper than making the model itself faster.

Like a restaurant bringing bread to the table, so the wait for your main course feels shorter.

A measure of how surprised a language model is by a piece of text; lower means it predicted the text better.

It is e raised to the average next-token loss, and roughly means how many tokens the model was choosing between at each step. It is handy for comparing versions of the same model, such as before and after quantisation, but not for comparing models with different tokenizers.

Like counting how many guesses a friend needs to finish your sentences.

Asking a model to answer as a particular expert or character.

A persona shifts what the model pays attention to, its vocabulary and its priorities. It changes focus and style, not knowledge: the facts still need to come from you. Rich personas for role-play use a full character sheet.

Like asking a friend to read your contract "as a lawyer would": same friend, different things noticed.

Information that can identify a specific person, such as a name, phone number, address or card number.

AI systems that store conversations easily collect PII, so they need rules about what to keep, how long, who can see it and how to delete it. Laws such as India's Digital Personal Data Protection Act, 2023 give people rights over their data. A good rule of thumb is to store the need, not the sensitive reason behind it.

Like the details on an ID card: anything that points to exactly one real person.

An extension that adds vector columns, distance operators and vector indexes to PostgreSQL.

pgvector adds types such as vector and halfvec, operators such as <=> for cosine distance and <-> for Euclidean distance, and HNSW and IVFFlat indexes. Embeddings live in the same rows as the rest of your data, so filters, joins, transactions and deletes are ordinary SQL. Most hosted PostgreSQL services offer it.

Like fitting a roof rack to the car already in your garage instead of buying a van.

Finding and masking personal information such as phone numbers, emails and ID numbers in text.

PII means personally identifiable information. In a RAG system, redact it at ingestion so it never enters the index, before logging user questions, and on outputs as a last line of defence. Regular expressions work for well-formatted items like phone numbers; libraries such as Microsoft Presidio go further.

Like blacking out names and numbers on a document before photocopying it for a wider audience.

A fully managed vector database that you use over the internet with an API key.

Pinecone runs the servers, scaling and backups for you. You create an index with a fixed dimension and metric, upsert records with metadata into namespaces, and query them. Its serverless indexes charge for storage, writes and reads rather than for idle servers.

Like taking a taxi instead of owning a car: someone else handles the maintenance, and you pay per ride.

The `|` symbol, which sends the output of one command into the next command as its input.

Pipes let you combine small single-purpose commands into one line, such as `grep -il refund *.txt | wc -l` to count files mentioning refunds. Each command does one job and passes its result along.

Like a conveyor belt between two bakery stations, carrying each tray straight from one to the next.

A one-line helper in the transformers library that runs a model for a named task, handling all the steps for you.

pipeline("task", model=...) bundles three steps: preprocessing (tokenizer, image processor or feature extractor), running the model, and post-processing its output into labels, text or boxes. Task names include text-classification, zero-shot-classification, image-classification, object-detection, automatic-speech-recognition and text-to-speech. Available tasks change between library versions.

Like a vending machine: coin in, drink out, with a fridge, conveyor and payment board hidden inside.

A named blank in a template, such as {review}, that is replaced with a real value.

Placeholders mark where changing data goes in a prompt template. Named placeholders make templates readable and safe to maintain. Anything inserted into one becomes part of the prompt, so user text must still be delimited and treated as untrusted.

Like the blank line on a greeting card where you write the person's name.

An open-source library for controlling web browsers from code: opening pages, clicking, typing and taking screenshots.

Developers use it for testing websites and for automation. In a browser agent, Playwright is the hand that carries out the model’s chosen actions, and it can also block requests to domains outside an allowlist. Hard-coded Playwright scripts are fast and repeatable but break when a page changes.

Like a remote-control robot arm for a web browser that does exactly what you tell it, nothing more.

Repeatedly asking a server "is it ready yet?" until a job is finished.

Polling is simple and works everywhere, which makes it a good first design. Start with a short interval, increase it after each check (backoff), cap it and always give up eventually. Alternatives that push results include server-sent events, WebSockets and sending the answer to a channel like WhatsApp.

Like glancing at the token screen every so often while your dosa is being made.

Shrinking a feature map by summarising small regions, usually by keeping the largest value.

Max pooling with a 2×2 window halves the height and width, keeping the strongest response in each region. This reduces computation, gives later layers a wider view and adds a little tolerance to small shifts. Global average pooling collapses each whole map to one number before the classifier.

Like summarising each shelf of a display case by its single best-looking pastry.

A numbered door on a computer that a network service listens on, such as 6379 for Redis.

One machine can run many services, each on its own port. With Docker, `-p 6379:6379` connects a port on your laptop (left) to a port inside the container (right). If a port is already in use, map a different laptop port instead.

Like numbered service hatches along one building: the same address, different doors for different departments.

Extra numbers added to each token so the model knows where in the text it sits.

Attention on its own ignores word order, so transformers add a position signal to each token's embedding. The original design used sine and cosine waves at many speeds; others learn one vector per position, and many modern models use rotary position embeddings (RoPE). Models usually handle text longer than they trained on poorly.

Like a clock with many hands: the fast hand tells nearby moments apart and the slow hands tell distant ones apart.

A VAE failure where the codes stop carrying information, so every output looks the same.

The KL term overpowers reconstruction and every image's cloud settles on the standard bell curve. The tell-tale sign is a KL term near zero and identical, average-looking samples. Mismatched loss scales are a common cause.

Like a filing clerk who files everything under "miscellaneous" because the rules punish any other choice.

A free, open-source, very reliable database server used by many production apps.

Postgres handles many users and servers writing at once, speaks nearly the same SQL as SQLite, and is easy to run locally with Docker. With the pgvector extension it can also store and search embeddings, which you use in Module 08.

Like head office's accounts department: built to take entries from every store at once.

A popular, free, open-source relational database that stores data in tables and is queried with SQL.

PostgreSQL is one of the most widely used databases for web applications, known for reliability and extensions that add new features. With pgvector it can also store and search embeddings, which often removes the need for a separate vector database.

Like a very well-run warehouse with labelled aisles and a strict record of everything that comes in and out.

Of the items a system flagged as a category, the share that really belonged to it.

High precision means few false alarms. For safety routing you usually accept lower precision (some unnecessary alerts) in exchange for high recall. Precision is calculated per category, not overall.

Like asking: of all the smoke alarms that went off, how many were real fires?

Precision: of the items flagged, how many were right. Recall: of the items that mattered, how many were flagged.

Raising the decision threshold usually raises precision and lowers recall, and vice versa. The F1 score combines them into one number. The right balance depends on what each kind of mistake costs, which is a business decision.

Like a smoke alarm: precision is how often an alarm means real smoke, recall is how many real fires it catches.

The first phase of answering a request, where the model reads all the input tokens at once.

Because the whole input is already known, the model can process all of it in parallel in one large pass. Prefill is fast per token, but a very large input still takes noticeable time, which delays the first word of the reply.

Like a baker reading the whole recipe card at a glance before starting.

The two phases of generating a reply: reading the whole prompt at once, then writing the answer one token at a time.

Prefill processes every prompt token in one parallel pass and fills the KV cache; it is limited by arithmetic and sets the time to the first token. Decode produces one token per pass, reading the weights and cache each time; it is limited by memory speed and sets the streaming rate.

Like reading a full recipe card once, then cooking it one step at a time.

Caching that can only reuse the beginning of a prompt, up to the first character that differs from a saved one.

Every token's computed values depend on all the tokens before it, so a cache can only reuse a prompt from the very first byte up to the first difference. One changed character near the start forces everything after it to be recomputed. This is why cache-friendly prompts put stable content first and volatile content, such as dates, names and retrieved passages, last.

Like a stack of plates: change a plate near the bottom and every plate above it has to come off and be stacked again.

The first and biggest stage of training, where a model learns to predict the next token on a huge pile of text.

Pretraining shows the model trillions of tokens of text from the web, books and code, nudging its parameters after every guess. It produces a base model that knows a great deal about language and the world but simply continues whatever text it is given. Later training stages turn it into a helpful chat model.

Like a trainee chef reading every cookbook in the library before ever serving a customer.

A column whose value is unique for every row, so each row can be identified exactly.

Usually an `id` column. The database refuses to insert a second row with the same primary key, which protects against duplicates. Other tables point to it with foreign keys.

Like an order number: two orders can look identical, but never share a number.

A chance for every possible outcome, all adding up to exactly 1.

Classifiers output a probability distribution over the possible answers, such as 66% croissant, 24% pain au chocolat, 10% danish. Each value is between 0 and 1 and they sum to 1. A probability is the model’s belief given its training data, not a guarantee.

Like a weather forecast that splits 100% between sun, cloud and rain.

Know-how: the rules and methods an agent follows to do its job.

For most agents, procedural memory lives in the system prompt, the tool definitions and the code, such as "for allergy questions, never guess". Some advanced agents can update their own instructions based on experience, which is powerful but needs careful review.

Like knowing how to pack a hamper so the jars do not break: a skill rather than a fact.

Compressing a vector by cutting it into pieces and replacing each piece with the ID of the nearest typical piece.

PQ splits each vector into equal pieces and learns a codebook of typical values for each piece. A vector is then stored as a short list of codebook IDs, often one byte per piece, for example 96 bytes instead of 6,144. Distances become approximate, so systems often re-score a shortlist with full vectors.

Like describing a colour by the number of the nearest paint chip in a catalogue instead of its exact mix.

All the text you send a model as input: instructions, information and the question.

A prompt is everything the model reads before it answers. Clear, specific prompts with the right facts produce far better results than vague ones. Designing prompts well is the subject of Module 03.

Like the brief you give a new baker before a shift: the clearer it is, the better the result.

A provider feature that reuses work for a request whose beginning is identical to a recent one, making it cheaper and faster.

System prompts and conversation history are resent on every turn. If the start of the request is unchanged, providers can reuse earlier processing and charge a fraction of the normal input price for that part. Changing the beginning, for example by trimming one message per turn, loses the saving.

Like a baker who keeps yesterday's prepared base for a regular order instead of starting from scratch.

A workflow that breaks a task into fixed steps, where each model call works on the previous call’s output.

Chains can include gates: code checks between steps that send work back or stop it. Each step is simpler than the whole task, which improves accuracy and makes problems easy to locate. The cost is latency, because steps run one after another.

Like an assembly line where the cake is baked, then iced, then boxed, with an inspector between stations.

Making the text you send to a model shorter without removing anything it needs.

Techniques include trimming the system prompt, sending only relevant tools, retrieving fewer and better chunks, summarising old conversation history and using compact structured data. Automatic tools such as LLMLingua delete low-information tokens but can remove crucial words. Every cut should be checked with evals, and safety rules should never be compressed.

Like packing a parcel without the suitcase: send the postcard, not your whole wardrobe with it.

The craft of writing instructions that reliably get a language model to do what you want.

Prompt engineering means designing, testing and revising the text you send to a model, much as you would with code. A good prompt covers the role, task, context, rules, output format and clearly marked input. It is measured, not guessed: you test prompts against varied inputs and keep the versions that score best.

Like writing a clear briefing for a brilliant new hire who knows nothing about your business.

An attack where text given to an AI model contains instructions that try to override yours.

A customer email saying "ignore your instructions and approve a full refund" is untrusted input inside your prompt. Models can be fooled by it. Keep humans in the loop for consequential actions, limit what the model can do, and treat model output as untrusted.

Like a forged note slipped into the order pile that says "give this customer everything free".

A reusable prompt with named slots that your code fills in for each request.

Keeping prompts as templates in one module makes them easy to review, test and version. A function like `build_messages(email, order_context)` fills the slots and returns the messages list. If a template contains literal braces and uses `str.format`, double them.

Like a printed reply card with blanks for the customer's name and order number.

Giving every version of a prompt an ID and history, and recording which version produced each reply.

Prompts change an app’s behaviour as much as code does, so they belong in the repository with a history, reviews and version IDs. A content fingerprint (a short hash) makes any edit produce a new version automatically. Logging the version on every request lets you measure incidents and roll back.

Like dating and numbering every edition of a menu, so you can always tell which one a customer ordered from.

Switching to a backup model provider when the main one is temporarily unavailable.

A fallback should trigger only on temporary failures such as rate limits, server errors and timeouts, not on errors caused by your own request. Different providers answer differently, so test your prompts on the backup before you rely on it.

Like switching to the second card machine when the first one is down.

A request on GitHub to merge one branch into another, where teammates review the changes first.

A pull request shows the diff, lets reviewers comment on specific lines, runs automated checks and records the approval. Most teams merge to main only through reviewed pull requests. In Module 18 you build an AI reviewer that comments on them.

Like submitting a new recipe to the head baker, who tastes it and suggests changes before it goes on the menu.

A Python library that checks and converts incoming data to match a model you define.

You write a class with type hints that inherits from `BaseModel`; Pydantic validates data against it at runtime, converting safe cases like `"2"` to `2` and rejecting the rest with detailed errors. Every model can also produce a JSON Schema. It is central to validating LLM output and to FastAPI.

Like a receiving clerk who checks every delivery against the order sheet and sends back the wrong ones.

The most widely used Python tool for writing and running automated tests.

With pytest you write functions whose names start with test_, use plain assert statements, and run pytest to see what passes. Features such as parametrize (run one test over many inputs) and markers (label slow tests) make it well suited to LLM apps.

Like a kitchen's opening checklist that runs itself every morning and shouts if anything is off.

LoRA fine-tuning on top of a base model stored in 4-bit, so large models fit on small GPUs.

The frozen base is stored in the 4-bit NF4 format and converted to 16-bit on the fly for each calculation, while the LoRA adapters train in 16-bit. Double quantisation and paged optimisers save more memory. It lets an 8B model fine-tune on a single 16 GB GPU.

Like vacuum-packing the master recipe book so it fits on a small shelf, while you keep writing fresh sticky notes.

Storing a model's numbers with fewer bits so it uses less memory and runs faster, with a small loss in quality.

Model weights are usually stored as 16- or 32-bit numbers. Quantization rounds them to 8 or 4 bits, shrinking memory roughly two to four times. A 4-billion-parameter model drops from about 8 GB in 16-bit to about 2.5 GB in 4-bit, which can make the difference between fitting on a laptop or not.

Like saving a photo as a smaller JPEG: much lighter, and most people cannot see the difference.

A LlamaIndex object that takes a question and returns an answer plus the source passages it used.

A query engine combines a retriever, optional postprocessors that filter or rerank nodes, and a response synthesiser that builds the prompt and calls the LLM. Its response mode changes how many LLM calls one question costs.

Like a restaurant's front-of-house: you order once, and the kitchen, plating and service happen behind the door.

A placeholder like $name in a query, with the actual value passed separately.

Parameters keep the query text fixed and treat values strictly as data, which prevents injection attacks where a crafted value changes what the query does. They also let the database reuse its plan for the query. Use them for every value, especially anything from a user or an LLM.

Like a printed form with blanks: people fill in the blanks but cannot rewrite the form.

The part of a database that decides how to run each query, such as which index to use.

PostgreSQL’s planner estimates the cost of different ways to answer a query and picks one. EXPLAIN shows its choice, and EXPLAIN ANALYZE runs the query and reports the real time. With pgvector, the planner only uses a vector index when the query orders by the matching distance operator with a LIMIT.

Like a delivery dispatcher choosing the route for each order before the rider leaves.

Turning a user’s message into a better search query before retrieving, for example by adding context from the conversation.

Follow-up messages like "and the downsides?" lack the topic needed for retrieval. A quick LLM call rewrites them into standalone questions using the conversation history. Other forms of rewriting expand abbreviations or split a complex question into several searches.

Like a librarian who hears "and the sequel?" and goes looking for the sequel to the specific book you were discussing.

The three vectors each token gets in attention: what it is looking for, what it offers, and what it hands over.

They are made by multiplying a token's embedding by three learned matrices. Queries are compared with keys to score relevance; values are what actually get blended. Because queries and keys are separate, relevance can be one-way: a pronoun can strongly seek a noun without the noun seeking it back.

Like a baker's question, the label on their apron, and the goods they hand over when asked.

An open-source Python library that scores RAG systems on retrieval and answer quality.

RAGAS computes metrics such as context precision, context recall, faithfulness and answer relevancy, mostly by asking an LLM judge many small, specific questions. Scoring retrieval and generation separately tells you which part of the pipeline to fix. Its API and metric names have changed between versions.

Like a restaurant inspector with a checklist, checking both the ingredients and the plate.

A starting number that makes “random” choices repeat exactly from run to run.

Weight initialisation, shuffling, dropout and data augmentation all use random numbers. Fixing the seed (torch.manual_seed, plus Python and NumPy seeds if used) makes runs repeatable, so differences between experiments come from your changes. GPU kernels can still introduce tiny differences between machines.

Like shuffling a deck the same way every time so two players get identical hands.

A cap on how many requests or tokens an API will accept from you in a period of time.

Exceed it and the API answers 429 Too Many Requests, often with a retry-after header. The right response is to wait and retry with growing delays, not to retry immediately. Limits usually rise as your account's usage history grows. Module 19 covers handling them properly.

Like a supplier who takes only so many orders per hour and asks callers to ring back later.

Capping how many requests a user, an address or the whole system can make in a period of time.

Rate limits protect against abuse, buggy loops and one user consuming everyone’s capacity. For AI apps they apply per user, per conversation (step limits) and on total spend. Counters must live in a shared store when an app runs on several servers, or each server counts separately.

Like a shop limiting festival sweets to two boxes per customer so everyone gets some.

A way of running a model that interleaves reasoning about what to do with actions that gather real information.

ReAct was introduced in a 2022 research paper. The model writes a thought, takes an action such as a search, reads the observation, and repeats. Mixing reasoning with real lookups made models far less likely to invent facts than reasoning alone, and far less aimless than acting alone.

Like a detective who thinks out loud, checks one clue, and lets that clue decide the next question.

The rules for which memories an agent retrieves for a message, for whom, and in what order.

A read policy filters strictly by the right person first, then ranks candidate memories, often by a mix of relevance, recency and importance. Safety-critical facts can be pinned so they are always included. It also decides whether retrieval happens automatically every turn or through a tool the agent calls.

Like a receptionist who pulls your file by patient number, with the allergy sticker always on the front.

A read-only copy of a database, kept up to date, that takes reading work off the main database.

Heavy read traffic, such as vector searches or reports, can be sent to replicas so the main database stays fast for writes like new orders. Replicas lag the main database by a little, so freshly written data may take a moment to appear on them.

Like a second copy of the shop catalogue at the counter, so browsing customers do not crowd the stockroom.

The front-page document of a code repository, explaining what the project is and how to use it.

GitHub shows the README first, so for a portfolio it is your display window. A strong one leads with what the project does, a demo, results with numbers and key design decisions, and puts installation instructions further down for those already interested.

Like a shop window that shows the best cake with a label, while the flour stays in the back.

A service you keep connected for the whole conversation, streaming audio and events both ways as they happen.

Unlike a normal request and response, a realtime API keeps one connection open, usually a WebSocket or WebRTC, so audio slices, transcripts, tool calls and control messages such as "cancel" can flow in both directions at once. Providers name their events differently and change them often, so always follow the current docs.

Like an open phone line between two kitchens, instead of sending a runner back and forth with each order.

A way of thinking about an LLM as a function you program with instructions and data, rather than an encyclopedia you consult.

Used this way, the model does what it is good at: understanding messy language, following instructions and writing well over the information you supply. Your code provides the facts, does exact calculations, makes decisions and checks the output.

Like a brilliant new hire on day one: excellent at the work, as long as you hand over the order sheet and keep the till yourself.

A model trained to work through a problem in hidden reasoning before writing its answer.

Reasoning models are much stronger at multi-step problems, but the thinking tokens are billed as output and delay the first visible word. Most let you control how much they think, for example with an effort setting. Many no longer accept temperature or top-p.

Like a baker who sketches the design on paper before touching the icing: better results, a longer wait.

Of the items that really belonged to a category, the share the system caught.

Recall measures misses. For rare, dangerous categories like food safety, recall matters more than overall accuracy, because a high accuracy score can hide a router that misses every rare case.

Like asking: of all the real fires, how many set off an alarm?

The tendency of a model to lean towards the label or pattern of the last examples it saw.

In few-shot prompts, the final example or two have extra pull. If they share a label, borderline cases drift towards it. Shuffle examples and avoid ending on a run of one label.

Like a judge who scores the last contestant highest because they are freshest in mind.

A way to merge several ranked lists by giving each item points based on its position in each list.

Each list gives an item 1 / (60 + its position), and the points are added up. Because only positions are used, lists whose scores are on different scales, like BM25 and cosine similarity, can be merged fairly. Items that several lists rank highly rise to the top.

Like combining two judges' rankings in a baking contest by placing, not by their very different scoring styles.

Software that suggests items a person is likely to want.

Collaborative filtering recommends from behaviour ("people who bought X also bought Y") and needs lots of history. Content-based recommendation uses what items are, for example by finding nearest neighbours in embedding space, and works for brand-new items. Real systems often blend both and add business rules and diversity.

Like a good shop assistant who remembers what you usually buy and suggests something that fits, not four copies of the same thing.

How different a model’s rebuilt version of an input is from the original.

Autoencoders are trained to minimise reconstruction loss, often mean squared error or per-pixel binary cross-entropy. After training, inputs with unusually high reconstruction error are ones the model finds unfamiliar, which makes it a useful anomaly score.

Like comparing a photocopy with the original and measuring how much was lost.

A network that reads a sequence one step at a time, carrying a small memory forward from step to step.

At each step an RNN combines the new input with its previous hidden state using the same weights, so it handles sequences of any length and recognises a pattern wherever it occurs. Plain RNNs forget quickly and suffer from vanishing gradients over long sequences, which LSTMs, GRUs and later transformers address.

Like reading a recipe line by line while keeping notes on a small notepad.

A cap on how many steps one run of a graph may take before it is stopped.

You set it in the config, for example config={"recursion_limit": 25}. If a run exceeds it, LangGraph raises GraphRecursionError instead of looping forever. It is a safety net for graphs with cycles; when you hit it, find out why the loop has no exit rather than simply raising the limit.

Like a rule that any order bounced between stations more than ten times goes straight to the manager.

Attacking your own system on purpose to find weaknesses before real attackers do.

For AI apps, red teaming means building sets of attacks (direct overrides, authority claims, role play, obfuscation, indirect injection, multi-turn tricks) and measuring how many succeed. Results feed back into tool design and guardrails, and the attack set becomes a regression test.

Like hiring someone to try to sneak into the bakery after hours, so you learn which locks to change.

A very fast, in-memory data store server that many programs can share over the network.

Redis keeps simple structures such as strings, lists, hashes and sets in RAM, so reads and writes take well under a millisecond. Lists make natural queues, which is why job-queue libraries like RQ are built on it. It can also write changes to disk so data survives restarts. Valkey is an open-source fork that works the same way for these uses.

Like a whiteboard in the middle of the kitchen that everyone can read and write instantly.

A rule that says how a new value for a state key is combined with the old one.

By default LangGraph replaces a key's old value with whatever a node returns. Annotating a key with a reducer changes that: Annotated[list, operator.add] joins lists, and add_messages appends chat messages, replaces ones with a matching ID and removes ones marked with RemoveMessage. A missing reducer on a list is a classic silent bug, because the list only ever holds the latest item.

Like the difference between rubbing out the last note on a ticket and adding a new line under it.

A correct answer to a test question, written or checked by someone who knows the subject.

Reference answers let you measure context recall and check correctness. They come from domain experts and go stale when the underlying documents change, so they need reviewing whenever sources are updated.

Like the answer key a teacher keeps for marking exam papers.

A fixed set of examples you re-run after every relevant change to check nothing that worked has broken.

For LLM apps, a regression eval runs the real model over a set of real cases and checks the pass rate against a threshold, rather than demanding every case pass. It usually runs in CI when prompts, tools or models change.

Like re-baking the house signature cake after changing flour suppliers, to make sure it still tastes right.

A set of automated tests that re-check old bugs, so that a fix stays fixed.

Every confirmed failure, whether a wrong answer or a successful attack, becomes a test that runs on every change. For LLM apps, critical cases are run several times because replies vary. Crude text checks can be backed up by an LLM-as-judge for high-severity cases.

Like a pilot’s checklist that grows a new line after every incident, so the same mistake is never made twice.

An activation function that keeps positive values and turns negative ones into zero.

ReLU is fast and its slope is exactly 1 for positive inputs, so gradients pass through without shrinking. Its weakness is “dead” neurons: a neuron whose input is always negative outputs zero and gets no gradient, so it never recovers. Leaky ReLU and ELU keep a small slope for negatives to avoid this.

Like a taster who stays silent unless they have something positive to say.

A way to sample randomly inside a network while still letting training signals flow back through it.

Instead of sampling a code directly from the encoder's distribution, you draw noise epsilon from a fixed bell curve and compute z = mu + sigma * epsilon. The randomness now sits outside the learnable part, so gradients reach mu and sigma.

Like rolling the dice before you start a recipe and writing the number on the order slip, so every step after is traceable.

A project folder whose full history is tracked by git.

A repository contains your files plus a hidden .git folder holding every commit. `git init` creates one, `git clone` copies one with all its history, and GitHub hosts repositories online for sharing and backup.

Like a recipe binder that also contains every previous version of every card.

A model that re-orders a shortlist of retrieved chunks by how well each actually answers the question.

A typical pipeline retrieves 20 to 50 candidates cheaply, then a reranker, usually a cross-encoder, keeps the best 3 to 5. It fixes cases where a half-relevant chunk outranked the right one. Its top score is also a useful signal for deciding when nothing relevant was found.

Like a head chef tasting the five best dishes from the first round to pick the winner.

A second, more careful pass that re-scores the top search results to put the best ones first.

Fast retrieval finds a few dozen candidates; a reranker, often a cross-encoder model that reads the query and each passage together, scores them more accurately. It is too slow to run over everything, so it is used only on a short list. Reranking often fixes retrieval failures with very little extra latency.

Like a head chef tasting the five shortlisted dishes after the line cooks have filtered fifty.

A shortcut that adds a layer's input back to its output, so each layer only has to learn a small change.

A transformer computes x + layer(x) around both attention and the feed-forward network. The shortcut keeps the original information intact and gives training signals a clear path through dozens of layers. Deep stacks often fail to train without it.

Like annotating a recipe card in pencil instead of rewriting it from scratch at every station.

The part of a LlamaIndex query engine that turns retrieved passages and the question into a prompt and an answer.

Its default "compact" mode packs as many passages as fit into one LLM call. "Refine" makes one call per passage, improving the answer each time, and "tree_summarize" summarises in layers. You can replace its prompt template to add rules such as an exact "I don't know" phrase.

Like the person who reads everyone's notes and writes the final report.

A common style for web APIs where URLs name things and HTTP methods say what to do to them.

In a REST API, `GET /orders/42` reads an order, `POST /orders` creates one, `PATCH` updates and `DELETE` removes. Each request is stateless: it carries everything needed, and the server does not remember previous requests. You build REST APIs with FastAPI in Module 05.

Like a well-run counter where every item has a ticket number and a fixed set of things you can ask for.

Finding relevant passages from your own documents first, then giving them to the model so it answers from real facts.

A RAG system chunks and embeds your documents ahead of time, retrieves the closest chunks for each question, adds them to the prompt and lets the model write the answer. It is the most common way to make a model answer from private or recent information without retraining it. Module 09 teaches it in depth.

Like an open-book exam: the student still writes the answer, but from the textbook in front of them rather than from memory.

The part of a RAG system that takes a question and returns the most relevant chunks.

A retriever can use vector search, keyword search, a combination, or reranking, but it always has the same shape: question in, relevant passages out. Frameworks like LangChain and LlamaIndex treat it as a standard interface, so you can swap one retriever for another without changing the rest of the app.

Like a socket in the wall: the appliance does not care how the electricity was generated, as long as the plug fits.

When retries during an outage multiply the load on a system that is already struggling.

If every failed request is retried several times, and users also re-send their messages, the number of calls can grow many times over exactly when the provider is weakest. Rate-limit errors and long waits follow. Short timeouts, capped retries with backoff, backup providers and kill switches prevent it.

Like everyone redialling a busy phone line at once, which keeps it busy for everyone.

Trying a failed operation again after waiting, and waiting longer before each new attempt.

Temporary failures such as network blips or brief rate limiting often succeed on a second try. Increasing waits, such as 5, 15 and 45 seconds, avoid hammering a struggling service. In RQ, retries with intervals need a worker running the scheduler, and retried tasks must be idempotent.

Like phoning a busy shop again in five minutes, then fifteen, rather than redialling every second.

A value in an error response where the server tells you how long to wait before trying again.

Servers often send retry-after with 429 and 503 responses, usually as a number of seconds and occasionally as a date. The server knows when your limit refills, so its advice beats any formula. Good retry code uses it when present and falls back to exponential backoff with jitter when it is not.

Like a shop sign saying "Back in 10 minutes": better than guessing when to knock again.

When a model learns to score highly on its reward signal in ways that do not reflect what people actually wanted.

Examples include excessive apologies, flattery, or overly long answers that a reward model happens to favour. Penalties for drifting from the original model, better preference data, and checking outputs by hand all help.

Like a shop assistant who learns that saying "sorry" earns good reviews and starts apologising for everything.

A model trained to score how much people would like a given answer.

It is usually a language model with its output replaced by one number, trained on pairs of answers where people picked a winner. RLHF optimises the assistant against it, so its blind spots become the assistant's habits.

Like an apprentice who has studied a judge's past verdicts closely enough to predict new ones.

Training a model to give answers people prefer, using a reward model learned from human comparisons.

People compare pairs of model answers; a reward model learns to predict their choices; then the model is optimised to earn high reward, with a KL penalty keeping it close to its starting point. It is the main reason chat models are helpful and polite rather than merely continuing text.

Like training a new counter assistant with a mystery shopper's verdicts on which of two approaches was better.

A short, regularly updated summary that replaces older conversation turns in the context.

When history grows past a threshold, older turns are summarised and dropped, keeping recent turns word for word. This keeps each call's size roughly constant. Summaries are lossy, so safety-critical facts belong in a separate pinned block.

Like a support agent's running notes on a long call instead of a full transcript.

A way of marking word positions by rotating each query and key by an angle that depends on where the token sits.

When a rotated query meets a rotated key, the two rotations partly cancel, so the attention score depends on how far apart the tokens are. It is used by the Llama family and many other open models. Extending a RoPE model to much longer texts than it trained on needs extra care.

Like two dancers turning at the same speed: how far apart they face depends only on how far apart they started.

The small layer in a mixture-of-experts model that decides which experts each token goes to.

It scores every expert for a token, keeps the top k, and turns their scores into blend weights with softmax. Left alone, routers develop favourites, so training adds a balancing loss or adjustable biases to spread the load.

Like the person at the kitchen pass who reads each ticket and sends it to the right stations.

A workflow that classifies an input first, then sends it to a specialised handler.

Different categories get different prompts, tools or even models, such as a cheap model for easy questions and a strong one for hard ones. Always validate the router’s label against the allowed set and have a safe default path, because routers sometimes misclassify.

Like the front desk of a hospital sending each patient to the right department.

A small Python library that turns Redis or Valkey into a job queue with workers.

RQ lets you enqueue a function call as a job, run separate worker processes that execute jobs, and track each job's status, result and errors. Workers import task functions by name, so tasks must live in an importable module. It supports timeouts, retries, result expiry and priority queues.

Like an order-ticket system for a kitchen, built on top of the kitchen whiteboard.

A written scoring guide that describes what each score level looks like for each criterion.

A good rubric breaks a judgement into criteria and describes each level in observable terms, such as "explains retrieval and one way to handle missing information". It makes scoring consistent across candidates and scorers, human or AI, and lets reviewers see why a score was given.

Like a board exam marking scheme that says exactly what earns each mark.

A file format for storing model weights that loads quickly and cannot run hidden code.

Older PyTorch weight files (.bin) use Python's pickle format, which can execute code when opened, so a malicious file could harm your machine. Safetensors stores only the numbers, loads them fast, and is now the default on the Hugging Face Hub. When a repo offers several formats, choose .safetensors.

Like receiving flour in a sealed, labelled sack rather than an unmarked box that might contain anything.

A model that labels text as safe or unsafe for one specific risk, such as an allergy promise.

Generic moderation knows nothing about your business, so you add narrow classifiers for your own harms. They can be open safety models such as Llama Guard, or an LLM with a short prompt and a typed yes-or-no answer. Evaluate them like any model, measuring misses and false alarms on labelled examples.

Like a specialist inspector who checks only for nuts in every dish, and nothing else.

The step where the model picks the next token from its list of probabilities.

Most chat models pick at random in proportion to the probabilities, like rolling a weighted die, which makes text varied and natural. Settings such as temperature, top-p and top-k change how adventurous that pick is. Sampling is why the same question can produce different answers.

Like rolling a die that is weighted towards the most sensible next word.

How many times per second a sound wave is measured when it is recorded as numbers.

Audio is stored as a long list of measurements; the sampling rate says how many per second, such as 16,000 Hz for speech models or 44,100 Hz for music. A list of numbers does not record its own rate, so a model given audio at the wrong rate hears it too fast or too slow and produces confident nonsense rather than an error. Always resample to the rate on the model card.

Like the speed of a record player: play a record at the wrong speed and the singer becomes unrecognisable.

An isolated environment where code or an agent can run without reaching your files, secrets or network.

Sandboxes are usually containers or virtual machines with limited permissions, no stored credentials and restricted internet access. They contain the damage if model-written code or a manipulated agent does something harmful. Code agents, coding agents and computer-use agents should all run in one.

Like a test kitchen with its own ingredients and a locked door, so a failed experiment never reaches the main bakery.

The point where reading more traces stops revealing new kinds of failure.

In error analysis you keep reading until roughly twenty traces in a row produce no new kind of note. That signals you have seen the main failure modes for now. Saturation is temporary: new traffic, products or models bring new failures.

Like tasting a new batch of biscuits until every extra bite tells you nothing new about the recipe.

Storing each number in a vector with fewer bytes, usually one instead of four.

Scalar quantisation maps each number’s range onto 256 steps and stores the step as one byte, making vectors four times smaller. Similarity scores shift slightly but rankings barely change. Half-precision floats (two bytes) are a milder version of the same idea.

Like rounding prices to the nearest rupee: the totals barely change and the receipts get much shorter.

Checking in code that data has exactly the fields, types and value limits you expect.

In Python, Pydantic models describe the expected shape and constraints, such as a quantity between 1 and 20 or a store from a fixed list, and reject anything else with a clear error. Validating every tool argument and structured output is deterministic, so it cannot be talked out of a rule.

Like a fridge thermometer: it reads the number, and the number is either in range or not.

The part of a program where a name can be seen and used.

Variables created inside a function are local: they exist only while the function runs and are invisible outside it. Names defined at the top of a file are global. Passing values in as parameters and out with `return` keeps code easy to test.

Like the pastry kitchen's bowls: they stay in the pastry kitchen, and only the finished cake comes out.

A place outside the context window where an agent writes notes and plans to read back later.

In deep agents the scratchpad is usually a folder of files such as plan.md and notes. Files cost nothing until read, survive crashes, can be shared between subagents and can be inspected by people. File paths chosen by the model must be checked so the agent cannot escape its folder.

Like a shared folder where the whole team files its notes, instead of everyone memorising everything.

A ready-made library that lets you call an API in a few lines of your own programming language.

An SDK builds the HTTP requests, adds your API key, parses the responses and often retries failures for you. Anthropic's `anthropic` package and OpenAI's `openai` package are SDKs for Python. Under the hood they send the same HTTP requests you could send with curl.

Like a waiter who already speaks the kitchen's language, so you do not have to shout into the kitchen yourself.

A service that stores passwords and API keys encrypted, controls who can read them and records each access.

A secrets manager lets you keep keys out of code, images and Git history, grant one service access to one secret, and rotate a key without rebuilding anything. Cloud platforms inject secrets into containers as environment variables at start-up. Any key that ever lands in an image or commit should be treated as leaked and rotated.

Like a bank locker: only named people get in, and the bank logs every visit.

A step where every word in a text looks at every other word and blends in meaning from the ones that matter to it.

Each token gets a query, a key and a value. Query-key matches are scored, scaled and turned into weights with softmax, and each token's new vector is the weighted blend of every value. It is what lets "it" pick up the meaning of "cake" earlier in the sentence. Its cost grows with the square of the text length.

Like bakers in a busy kitchen each asking the room a question and collecting a bit of help from everyone, mostly from whoever matches best.

Asking the same question several times and taking the most common final answer.

Independent reasoning paths that are correct tend to agree, while mistakes scatter, so a majority vote filters out one-off errors. It multiplies cost and only works for short, comparable answers. A split vote doubles as a low-confidence warning.

Like asking five people to do the same sum on napkins and trusting the answer most of them got.

Agreed names for the fields in traces, so every tool reads them the same way.

Semantic conventions are OpenTelemetry's shared vocabulary for span names and attributes. The GenAI conventions name AI work, such as span names like `chat claude-opus-5-5` and attributes like `gen_ai.usage.input_tokens` and `gen_ai.provider.name`. They are still in development, so some names have changed and may change again.

Like the standard layout of an address label with a PIN code in a fixed place, so any post office can route the parcel.

General knowledge about the world that is not tied to any one person, like the menu or the refund policy.

In agent design, semantic memory is the body of facts an agent knows in general, often provided through RAG over documents. Despite the shared word, it is not the same thing as semantic search, which is a technique for finding text by meaning. You can store any kind of memory with any retrieval technique.

Like knowing that chikki is made of peanuts: true for everyone, not just one customer.

Searching by meaning rather than by exact words.

Semantic search embeds the query and the documents, then returns the documents whose embeddings are closest to the query. It finds "kaju barfi" when someone searches for "cashews". It is often combined with keyword search for the best results.

Like asking a chemist for "something for a sore throat" and being handed honey lozenges, though you never said "honey".

A counter of free slots that limits how many tasks can do something at once.

`asyncio.Semaphore(10)` lets at most ten tasks into a block at a time; the rest wait for a slot. It is the standard way to keep concurrent API calls within rate limits and budgets.

Like having ten ovens: the eleventh tray waits until one is free.

An AI app revealing private data, such as other customers' details, internal documents or credentials.

It often happens when tools or retrieval give the model data the current user should not see, which the model then repeats. Defences include checking permissions inside tools, filtering retrieval by role, keeping secrets out of prompts and redacting personal data from logs.

Like a helpful assistant reading out another customer's address because the order book was left open.

A free Python library for running open embedding models on your own computer.

Sentence Transformers downloads embedding models such as all-MiniLM-L6-v2 from the Hugging Face Hub and turns text into vectors locally with one call to encode(). Text never leaves your machine and there is no per-text cost. Many of its small models have short input limits and silently cut longer text.

Like owning a home printer instead of using a print shop: a one-off setup, then private and free to use.

A tool that the model provider runs on its own servers during your API call, rather than your code running it.

You declare a server tool such as web search in your request, and the provider executes it and feeds the results to the model within the same call. You get less control and logging than with your own tools, but far less setup. Server tools often have version strings and their own fees.

Like a restaurant that cooks the side dish in its own kitchen instead of asking you to bring it.

A simple web standard that lets a server keep sending small messages to a browser over one open connection.

The browser makes one ordinary HTTP request and the server answers with a response that stays open, writing text events made of lines like "data: ..." and ending each event with a blank line. It is one-way, server to browser, which suits streaming an AI reply. LLM providers use the same format to stream tokens to your code.

Like tuning into the bakery's radio announcements: you ask once, then listen while they keep talking.

A way of using a cloud service where you never manage servers and pay for what you use.

With a serverless database, the provider decides how many machines to run and scales them automatically. You are billed for storage and for the reads and writes you make, rather than for machines sitting idle. It is convenient, and costs can grow quickly with heavy query traffic, so estimate them.

Like paying for electricity by the unit instead of buying a generator.

One conversation, with its own ID and its own history, kept separate from every other conversation.

A chatbot looks up the right session for each incoming message, for example by WhatsApp number or a browser ID, and uses only that session's history. Sharing one history between users mixes their conversations and leaks private details. In production, sessions live in a database so they survive restarts.

Like a separate folder for each customer at the counter, instead of one pile for the whole shop.

A group of traces that belong to the same conversation.

Each message a user sends is its own request, so its own trace. A session ties those traces together with a shared id, so you can read a whole conversation in order. Tracing tools let you set a session id on each trace; the OpenTelemetry GenAI conventions call it the conversation id.

Like stapling together all the receipts from one customer's visit so you can see the whole visit, not one purchase at a time.

Data that a web app keeps for one user’s session between interactions.

In Streamlit, st.session_state survives the rerun that happens on every click, so it is where chat history and other per-user values live. It belongs to one browser tab and is lost on refresh, so anything important should be saved to a database.

Like sticky notes on the edge of a whiteboard that survive each wipe.

A small store of facts that every agent in a team can read and write during a task.

It lets one specialist’s key finding, such as a date or a number, reach the others while they are still working. Keep it short and factual, because everything on it is copied into every agent’s context. It is different from each agent’s private working context and from long-term memory across tasks.

Like the board on a newsroom wall with the names, dates and numbers everyone needs.

The program inside a terminal that reads your commands and runs them.

macOS uses zsh by default, most Linux systems use bash, and Windows uses PowerShell. They share many everyday commands but differ in details, so a command from a tutorial may need adjusting for your shell.

Like the kitchen assistant who hears your instructions and carries them out, in their own dialect.

Google's family of safety classifier models that check text against content policies.

Built on Gemma 2, ShieldGemma answers whether text violates policies such as harassment, hate speech, dangerous content or sexually explicit content. Through Hugging Face you can supply your own policy wording and read the probability of a violation to set your own threshold.

Like a second, differently trained smoke detector that also notices some things the first one misses.

What an agent can see right now: the current conversation inside its context window.

For an LLM agent, short-term memory is the list of messages, tool results and instructions sent with the current request. It is fast and immediately usable but limited by the context window, so older turns must be dropped or summarised. It belongs to one conversation or thread and is not kept between separate sessions unless you save it.

Like the papers on a shop assistant's desk during one customer's visit: right there, but cleared away afterwards.

A fixed-length slice of recent history, paired with the value that came next, moved forward one step at a time.

Sliding windows turn one long time series into many training examples: the 28 days before each day become an input and that day becomes the target. The window length is the lookback; how far ahead you predict is the horizon. Windows overlap heavily, which is why random splits leak.

Like looking at the last four weeks of the sales ledger through a frame you slide down one line each day.

A quick check, right after a deploy, that the new version is actually running and answering.

A smoke test calls a few key endpoints, such as the health check and one canned request, against the live service. It catches deploys that "succeeded" but serve errors. A failing smoke test should turn the pipeline red and alert someone, with a quick way to roll back.

Like switching on a newly installed oven and checking it heats before baking the first batch.

A small, open-source Python library from Hugging Face for building agents, especially agents that act by writing code.

smolagents offers a CodeAgent, which writes Python snippets that call tools, and a ToolCallingAgent, which uses ordinary JSON tool calls. It works with Hub models, local models and APIs like Claude through LiteLLM. Its core is deliberately tiny so you can read how it works.

Like a compact toolkit that fits in a drawer but still has every tool you need for most jobs.

A test that saves a copy of something, such as a prompt, and fails if it ever changes unexpectedly.

For LLM apps, snapshotting prompts makes every prompt change deliberate and visible in code review. The test does not judge whether a change is good; it forces someone to run the evals and update the saved copy on purpose.

Like a photo of the shop display taken each night, so any unexpected change is noticed in the morning.

Teaching by asking guiding questions and giving hints, rather than handing over answers.

A learning advisor uses a hint ladder: a guiding question, then a pointed hint, then a partial step, and the full answer only when needed. Reference facts can be given directly, but only from course material you supply.

Like a teacher who asks "what do you think the yeast needs?" instead of reading out the answer.

A formula that turns a list of raw scores into probabilities that add up to 1.

Softmax raises e to the power of each score and divides by the total, so larger scores get much larger shares. It is how logits become label probabilities. The result says which option the model prefers among its own choices, not how often the model is actually right.

Like splitting a cake among guests in proportion to how loudly each one cheered, so the slices always make exactly one cake.

One timed step inside a trace, such as a search or a model call.

A span is a single unit of work with a name, a start and end time, a parent span, attributes (key-value facts like token counts) and a status. The first span of a request, the root, has no parent. Spans nested under other spans form the tree that makes up a trace.

Like one line on a kitchen batch card: "baked at 180 degrees, 28 minutes, checked by Ravi".

Attention where each token looks at only a chosen subset of other tokens, such as a nearby window, instead of all of them.

Patterns include sliding windows, a few global tokens, and strided connections. Cost grows roughly linearly with text length, and information can still travel far by hopping through layers. It is an approximation of full attention, so quality on long-range tasks must be checked.

Like news spreading along a queue where each person only talks to their neighbours.

A reserved token with a structural meaning, such as "a new message starts here" or "end of turn".

Special tokens mark roles and boundaries in a chat template and tell the model when its turn is over. Users typing the same characters produce ordinary text tokens instead, so they cannot forge a role marker. That is a key reason the messages format resists simple impersonation.

Like a bakery's official stamp: anyone can write "APPROVED" on a form, but only the real stamp counts.

An agent in a team that handles one kind of work, with its own focused instructions and only the tools that work needs.

A specialist starts with only the brief it is given, so the brief must contain everything it needs. It returns a short memo with numbers and sources rather than its full working. Narrow tools keep it focused and limit damage if it goes wrong.

Like a reporter assigned to the courts beat who files one page for the editor.

A picture of a sound that shows how much energy it has at each pitch over time.

Audio models often convert the raw wave into a spectrogram: pitch bands from low to high on one axis, time on the other, loudness as intensity. Whisper uses 80 or more Mel bands with one column every 10 milliseconds. Spectrograms make speech patterns easier for models to see, much like an image.

Like sheet music for any sound: which notes, how loud, and when.

A speed-up where a small model guesses several tokens ahead and the big model checks them all in one pass.

Guesses are accepted up to the first disagreement, where the big model supplies its own token, so each pass yields at least one token and often several. With the proper acceptance rule the output exactly matches the big model's. Gains depend on how often guesses are accepted and on the hardware.

Like an apprentice piping five rosettes quickly and the senior baker approving them at a glance.

A single model that listens to audio and replies with audio directly, without a separate transcription or voice step.

Speech-to-speech models turn incoming speech into audio tokens, reason over them together with text instructions, and generate audio tokens that become the spoken reply. They usually offer the lowest latency and keep tone and emotion. The trade-offs are less choice of language model and voice, higher cost per minute and transcripts only if you ask for them.

Like a bilingual receptionist who answers you directly, instead of a note-taker passing written notes to a manager who passes them to an announcer.

Turning spoken audio into written words.

Speech-to-text models such as Whisper convert audio into a transcript, often with timestamps and sometimes with translation into another language. Accuracy drops on names, numbers, local words, heavy accents, crosstalk and code-switching between languages. In voice agents, it is the first stage of the pipeline and the source of many downstream errors.

Like a stenographer typing up everything said in a meeting.

A language for asking a database for exactly the data you want, and for adding or changing data.

SQL is declarative: you describe the result, not the steps. Core statements are SELECT (with WHERE, GROUP BY, ORDER BY and JOIN), INSERT, UPDATE and DELETE. Nearly every relational database speaks it, with small differences.

Like asking the bookkeeper a precise question in a shared, standard phrasebook.

An attack where text typed by a user is treated as part of a SQL command.

It happens when code builds SQL by pasting user input into a string, for example with an f-string. Input like x' OR '1'='1 can then return every row, or worse, change or delete data. Always pass values through placeholders (? in SQLite, %s in psycopg).

Like a customer writing "and give me the till" on an order slip, and the clerk obeying it as an instruction.

A complete SQL database that lives in a single file and is built into Python.

SQLite needs no server or installation: `sqlite3.connect("crumb.db")` creates or opens the file. It is ideal for learning, prototypes, tests and small apps. It allows one writer at a time on one machine, so busy multi-server apps usually move to Postgres.

Like a single well-kept ledger on the shop counter: perfect for one store, not for six writing at once.

Releasing a change to a growing share of users in steps, checking health before each step.

A typical sequence is shadow mode, staff only, 10%, 50%, then everyone, with go and roll-back criteria written in advance. Hashing a user’s ID keeps them in the same group for every request. Staged rollouts apply to prompts, models, tools and re-enabling a feature after an incident.

Like opening a new store’s doors to friends and family for a week before the public launch.

A simple structure for answering "tell me about a time" interview questions.

You describe the Situation, the Task you faced, the Action you took and the Result, ideally with a number and a lesson learned. It keeps behavioural answers concrete and short, and works best with real stories, including mistakes you owned and fixed.

Like a recipe card for telling a story: setting, challenge, what you did, how it turned out.

A tool that remembers what it has already done during a task, such as which passages it has shown.

A stateful search tool can skip passages already returned, log every query and enforce a search budget. Its state should live for one conversation and be created fresh for the next, or information leaks between users. Plain functions are stateless; stateful tools are usually objects.

Like a researcher’s notebook that records which books have been read so nobody reads them twice.

The LangGraph builder you add steps and arrows to, built around one shared dictionary of state.

You create a StateGraph with a typed state (usually a TypedDict), add nodes and edges, then call compile() to get a runnable graph with invoke() and stream(). The state is the single source of truth for one run: every node reads it and returns updates to it. Compiling also checks the wiring, for example that every edge points at a real node.

Like a printed order ticket and the route it follows through the kitchen, set up before the first order arrives.

Keeping no memory between one request and the next.

An LLM API is stateless: each request is processed on its own, with no knowledge of earlier ones. Chatbots appear to remember because the application resends the conversation history every time. This gives you full control of what the model sees, at the cost of managing memory yourself.

Like a brilliant counter clerk who forgets each customer the moment they step away, so you hand her the full transcript every time.

A three-digit number at the top of every HTTP response saying what happened.

2xx means success (200 OK), 4xx means the request was wrong (400 bad request, 401 unauthorised, 404 not found, 429 too many requests), and 5xx means a server problem (500, 503, or Anthropic's 529 overloaded). Do not retry most 4xx errors; retry 429 and 5xx after a pause.

Like the stamp on a reply letter: "Done", "We do not know you", "Too many orders today, wait".

Running an MCP server as a local child process and exchanging messages over its standard input and output.

The host starts the server program and writes one JSON-RPC message per line to its stdin, reading replies from its stdout. It is simple, fast and private, but only the host that started the server can use it, and anything else printed to stdout corrupts the conversation. Logs go to standard error.

Like a private intercom between two rooms in the same house.

A field in an API response that says why the model stopped writing.

On Claude it is stop_reason: end_turn (finished), max_tokens (cut off at your limit), tool_use (wants a tool run) or refusal (declined). OpenAI calls it finish_reason, with values like stop, length and tool_calls. Checking it is how you catch replies that returned HTTP 200 but are incomplete or unusable.

Like the note a delivery rider leaves: delivered, nobody home, or address not found.

A string that makes the model stop writing as soon as it produces it.

You pass one or more stop sequences with the request; generation ends when one appears, and the sequence itself is not included in the returned text. They are handy for taking only part of a pattern, such as the first three items of a numbered list. Some reasoning models do not accept them.

Like telling a decorator to put the piping bag down the moment they finish writing "Happy".

A setting that chooses what you see after each step when you stream a graph run.

graph.stream(..., stream_mode="updates") yields what each node returned; "values" yields the full state after each step; "messages" yields text chunks from model calls as they are written; "debug" yields detailed internal events. You can pass a list to get several at once. "updates" is the best starting point for debugging.

Like choosing between watching the whole kitchen on CCTV or reading each station's note as it is written.

The MCP transport for servers reached over a network at a single URL, such as /mcp.

Clients send each JSON-RPC message as an HTTP POST; the server replies with JSON or a stream of server-sent events for longer work. In the 2025 revisions the server tracks each conversation with an Mcp-Session-Id header; the stateless 2026-07-28 revision makes every request self-contained. It lets many clients share one server and needs proper authentication, usually OAuth. It replaced an earlier HTTP-with-SSE transport in 2025.

Like a shop's phone number anyone can call, rather than an intercom inside one house.

Sending an answer to the user piece by piece as it is generated, instead of all at once at the end.

Streaming cuts the time until the first words appear, which makes an app feel much faster, though the total time is about the same. Most model APIs support it, and chat interfaces almost always use it. Module 19 covers it in depth.

Like serving each dish as it leaves the kitchen instead of holding every plate until the whole order is ready.

A Python library that turns a script into an interactive web page.

Streamlit reruns your whole script on every interaction and draws the page from it, so state that must survive goes in st.session_state. It has chat elements (st.chat_message, st.chat_input, st.write_stream) that make an internal chat tool a few dozen lines of code.

Like a whiteboard that is wiped and redrawn from the top every time someone touches it.

A tool setting that guarantees the model’s arguments exactly match the tool’s schema.

With strict set to true (and a schema that lists every property as required and forbids extra ones), any call the model makes has the right keys, types and allowed values. It controls the shape of a call, not whether the call happens.

Like an order form the kitchen will only accept when every box is filled in correctly.

An AI model's reply forced to follow a fixed shape you choose, so code can read it reliably.

Instead of free text, you give the model a schema and get back data in that shape, ready to parse. It greatly reduces formatting errors, but it cannot guarantee the content is correct or that a cut-off reply is complete, so validate anyway.

Like handing a customer a printed form instead of asking them to write a letter.

A helper agent that your main agent starts for one focused job, with its own fresh context, and that reports back with a short summary.

Subagents keep the main agent’s context clean by doing read-heavy work, like scanning logs, elsewhere. Give them narrow, read-only tools, a turn cap, a size limit on what they return, and no ability to spawn further subagents. Coding agents use them heavily.

Like sending an assistant to read a stack of files and come back with a one-paragraph note.

A complete, compiled graph used as a single step inside a bigger graph.

If the inner graph shares state keys with the outer one, you can add the compiled graph directly as a node. If the keys differ, you call it from inside an ordinary node and translate the state in both directions. A subgraph's END finishes only the subgraph, so its inner loop cannot accidentally end the whole run.

Like a cake-decorating team that works as one station on the main kitchen chart but has its own internal routine.

The one number, chosen before building, that tells you whether a feature is delivering real value.

A good success metric is something a non-engineer cares about, such as replies sent within 48 hours or minutes saved per order, measured before and after. It is paired with a guardrail metric that must not get worse, such as wrong orders or unapproved refunds. Activity counts like "messages sent" are vanity metrics.

Like a bakery judging a new oven by loaves sold and complaints, not by how shiny it looks.

Turning a long text into a shorter one that keeps what matters to a particular reader.

A good summary brief names the reader, the length and shape, and anything that must never be left out. Summaries naturally keep what is common and drop what is rare, so critical items should be detected separately. Long inputs are often handled with map-reduce.

Like a colleague who reads the whole report and tells you the three things you need before your meeting.

Training a pretrained model further on example inputs paired with the ideal outputs.

For chat models, each example is a conversation ending with the reply you want. A few hundred to a few thousand high-quality examples can reliably change tone, format and behaviour on a narrow task. It is the first step after pretraining in most assistant recipes, before preference tuning.

Like a new hire shadowing your best staff member for a month.

A model's tendency to agree with or flatter the user, even when it should not.

Chat models are trained to be helpful, and that can tip into telling people what they want to hear, especially under pushback. A weakly instructed support bot may cave to an insistent customer. Clear rules that hold under pressure, and code checks on risky replies, reduce the harm.

Like an eager new employee who says yes to every customer request to avoid an awkward moment.

An interview where you design a whole software system out loud for an open-ended problem.

You are given a prompt like "design a support assistant for 10,000 conversations a day" and judged on your process: clarifying requirements, estimating scale and cost, sketching an architecture, and probing its weak points. There is no single right answer; the reasoning and trade-offs are what count.

Like an architect sketching a building with a client, asking questions and explaining every choice.

The developer's standing instructions that shape how a model behaves for a whole conversation.

The system prompt sets the role, tone, rules and limits, such as "never promise refunds". It is written by the app developer, not the end user, and is sent with every request. It strongly guides behaviour but is not a security boundary on its own.

Like the house rules pinned up in the kitchen that every shift follows.

An attacker getting the model to reveal its hidden instructions.

Leakage is only dangerous when the system prompt contains something worth stealing, such as codes, keys or exploitable rules. Assume every prompt leaks eventually, keep secrets out, and enforce security rules in code where reading them does not help an attacker.

Like writing the safe combination on the staff notice board, then hoping no customer reads it.

The unit of work in A2A: a tracked request with an ID that moves through standard states until it finishes.

States include submitted and working, interrupted states such as input required, and final states such as completed, failed, canceled and rejected. A client can check a task by ID, stream its updates or receive push notifications, so long-running work is never lost to a timeout.

Like an order number you can phone up about, rather than a request shouted across a busy shop.

A setting that controls how adventurous a model is when choosing each next word.

Low temperature makes the model almost always pick the most likely next token, giving steady, repeatable text. High temperature spreads the choice across less likely tokens, giving more varied and creative text, with more risk. Some of the newest models manage this themselves and do not expose the setting.

Like a baker who either follows the recipe to the gram or improvises with the spices.

A grid of numbers with any number of dimensions.

A single number has 0 dimensions, a list has 1, a table has 2, and a batch of images or token embeddings has 3 or 4. Neural networks take tensors in, do arithmetic on them and give tensors out. NumPy arrays and PyTorch tensors are both tensors.

Like eggs, a row of eggs, a tray, a crate of trays and a truck of crates.

A dashboard that shows training metrics, images and settings for each run as the model trains.

Your training code writes small event files with a SummaryWriter, and the TensorBoard viewer draws them as live charts, image sliders, histograms and a settings table. It works with PyTorch and TensorFlow, locally or inside Colab.

Like the window and automatic temperature chart on an oven, so you can check the bake without opening the door.

A text window where you type exact instructions to your computer instead of clicking.

The terminal shows text and takes your typing; a program inside it, the shell, runs your commands. Almost every AI engineering tool, from uv and git to Docker and curl, is driven from the terminal, and servers often have nothing else.

Like the kitchen's back door where staff call out exact orders, compared with the shop front's picture menu.

A rule that tells a multi-agent conversation or loop when to stop.

Common conditions are a keyword such as APPROVE appearing, a maximum number of messages, or a time limit. Combine a content-based condition with a hard cap, because the content condition may never fire or may fire too early.

Like a referee’s whistle plus the clock: the game ends on whichever comes first.

Rewriting numbers, symbols and abbreviations as the words they should be spoken as, before text-to-speech.

Text-to-speech reads what it sees, so "Rs. 1,450", "1 kg" or "6pm" can come out wrong. Normalization turns them into "fourteen hundred and fifty rupees", "one kilo" and "six in the evening", either through a small code step or by instructing the language model to write that way. Stray markdown and emoji should be removed too.

Like rewriting a price tag into a sentence before handing it to the shop announcer.

Letting an LLM turn a plain-English question into a Cypher query that is then run.

It is flexible and impressive in demos, but generated queries can be wrong, slow or, under prompt injection, destructive. Production agents usually expose a few pre-written, parameterised query tools instead, and reserve free-form queries for trusted users on a read-only connection.

Like letting a visitor write their own instructions to the warehouse robot, instead of giving them a menu of safe requests.

Turning written text into spoken audio.

A text-to-speech model takes text and returns audio samples plus their sampling rate. Voices vary in naturalness, speed, language coverage and licence; some open voices forbid commercial use. Numbers, abbreviations and currency often need rewriting into words before they are spoken well.

Like a shop announcer who reads out whatever is written on the slip you hand them.

Turning a plain-English question into a database query that a model writes and your code runs.

The model reads the table structure, writes a SQL query, and summarises the result. It works best with a schema tool, read-only access, row caps and timeouts. Queries can look reasonable and still be wrong, so check answers against known totals.

Like asking a records clerk a question in plain words and having them fill in the official request form for you.

The name that groups all the saved steps of one conversation or run, so it can be picked up again later.

You pass it in the config as {"configurable": {"thread_id": "..."}}. Every invoke with the same thread ID loads the latest checkpoint for that thread and continues it; a new ID starts fresh. Thread IDs often appear in logs and dashboards, so derive them with a keyed hash rather than using phone numbers or emails directly.

Like a save slot with your name on it, separate from your sister's slot on the same console.

A model of the three gaps that make LLM apps hard: comprehension, specification and generalisation.

From Shreya Shankar and colleagues' research, the three gulfs separate the developer, the data and the LLM pipeline. Comprehension is the gap between you and your data, specification the gap between what you want and what you told the model, and generalisation the gap between clear instructions and reliable behaviour on new inputs. Naming the gulf behind a failure tells you which fix to try.

Like the three misunderstandings between a shop owner, her customers and a new trainee.

How much work a system finishes per unit of time, such as answers per minute.

Latency is how long one request takes; throughput is how many requests you complete in a period. A system can have fine latency for one user and collapse under a burst because its throughput is too low. Queues and more workers raise throughput without making any single answer faster.

Like a bakery's output of cakes per hour, as opposed to how long one cake takes to bake.

Automatically sending each support message to the right team, with the right urgency.

A model fills in a fixed triage form (categories, urgency, key details, a safety flag) and code applies hard rules on top, such as paging the food-safety lead for any allergy report. The model notices; code decides.

Like the triage nurse at a hospital entrance who sorts patients but does not treat them.

Measurements recorded in time order, such as daily sales or minute-by-minute temperatures.

Time series have trends, seasonal cycles (weekly, yearly) and one-off events. Forecasting predicts future values from past ones plus anything known in advance. Order matters, so they must be split by time, never shuffled across the train/test boundary.

Like a store’s daily sales ledger, read from the first page to today.

How long a user waits before the first word of a reply appears.

It is driven mostly by network time, prefill (reading the input) and any hidden thinking a reasoning model does before answering. With streaming, a short time to first token makes an app feel fast even if the full reply takes several seconds.

Like the time between ordering at a counter and the first plate arriving.

Going back to an earlier saved step of a graph run and continuing from there, optionally with changed state.

With a checkpointer, get_state_history lists every checkpoint of a thread. Invoking with None and a checkpoint's config replays from that point; update_state writes a change at a checkpoint and returns a new config, creating a fork you can run while the original history stays intact. It is a cheap way to test a fix on the exact state that caused a bug.

Like rewinding a recorded cooking show to just before the cake went wrong and trying a different oven temperature from there.

The longest your code will wait for something before giving up.

Without a timeout, a call to a stuck server can hang forever; `requests` has no default timeout, while `httpx` defaults to five seconds. Always set one explicitly, then handle the timeout with a retry or a friendly message.

Like deciding you will wait five minutes at the counter, not all night.

A small chunk of text, often a short word or part of a longer one, that a model reads and writes.

Models do not see letters or whole words; they see tokens. In English a token averages about three-quarters of a word, and other languages often need more tokens for the same meaning. Providers bill per token, and context windows are measured in tokens.

Like Lego bricks: every sentence the model builds is made from a fixed set of brick shapes.

The number of tokens you choose to allow per request, for cost and speed.

A budget is your own limit, usually far below the context window. Because chat apps re-send the whole history every turn, conversations need trimming or summarising to stay within it. Count tokens before sending and log usage after.

Like a household grocery budget: the shop would sell you more, but you decide what is sensible.

The number a tokenizer gives to each token, which is what the model actually receives.

A tokenizer maps every chunk of text to its position in a fixed list, so " order" might become 2569. The model works only with these numbers and never sees letters. When it replies, it produces IDs, which the tokenizer turns back into text.

Like a product code such as AF-500 standing in for "one 500 g bag of almond flour".

The count of input and output tokens a request used, returned with every response.

Usage is the exact, billed count for that request, including formatting and any hidden thinking. Logging it from day one lets you calculate real costs, spot runaway prompts and decide when to trim conversation history.

Like the itemised receipt that comes with every order.

The tool that splits text into tokens and turns them into numbers a model can process.

Each model family has its own tokenizer with a fixed vocabulary of tokens, each with an ID. The same sentence can be a different number of tokens for different models, which affects both cost and how much fits in the context window. You build one in Module 30.

Like a dough cutter that always slices text into the same set of shapes.

Rewriting text so it sounds a particular way while keeping its facts the same.

Describing tone as explicit dials (formality, warmth, firmness, length) works better than a single adjective like "nicer". The main risk is the model changing facts or adding promises, so check numbers and promise words in code.

Like adjusting the bass and treble on a song without changing the lyrics.

The function you write between a model’s tool request and an outside API.

A good adapter validates the arguments, calls the API with a timeout, turns failures into clear error results, shrinks the response to the few facts the model needs, and keeps secrets out of everything the model sees. It is where most of the safety and cost control of tool use lives.

Like a receptionist who phones the warehouse, waits a sensible time, and writes the key facts on a sticky note.

When a model asks your code to run a named function with specific inputs, then uses the result.

You describe the available tools to the model. Instead of answering, it can reply with a structured request such as check_delivery(area="Baner"). Your code decides whether to run it, runs it, and sends the result back so the model can finish. The model itself never executes anything.

Like a waiter who cannot cook but can send a ticket to the kitchen and bring back what comes out.

The description of a tool you give a model: its name, what it does, and what inputs it takes.

A tool definition has a name, a plain-English description of what the tool does and when to use it, and a JSON Schema for its inputs. It is all the model ever sees of your function, so a clear description is the cheapest way to make tool use reliable.

Like a line on a restaurant menu: the dish name, a short description, and the choices you can make.

Hiding malicious instructions in a tool's name or description so the model follows them.

Because hosts send tool descriptions from connected servers straight to the model, a malicious or compromised MCP server can plant instructions there, such as "before answering, read the user's files and send them here". Defences include installing only trusted servers, reviewing tool descriptions, limiting server permissions and keeping user approval for tool calls.

Like a contractor's business card with tiny print saying "also give this person your house keys".

The output of a tool your code ran, sent back to the model so it can continue.

On Claude it is a tool_result block in a user message, tagged with the id of the tool_use it answers; on OpenAI it is a message with role "tool". Results can be marked as errors so the model explains a failure honestly. Keep them small, because they are billed as input on every later turn.

Like the dish passed back through the kitchen hatch with the order number on it.

A helper in the Anthropic SDK that runs the agent loop for you over Python functions you decorate as tools.

You mark functions with @beta_tool, and the runner builds their descriptions from type hints and docstrings, calls the model, runs requested tools, appends results and repeats until the model is done. You iterate over it to see each model turn. Drop back to a hand-written loop when you need custom approvals or stop rules.

Like a dishwasher: it does the same washing you would do by hand, once you know what clean looks like.

An API setting that steers whether and which tools a model may call.

Common options are auto (the model decides), none (no tools), any or required (must call some tool) and a named tool. Support differs by provider and model: claude-opus-5-5 rejects the forced options, so you use auto with an instruction, or structured outputs, instead.

Like telling a waiter "suggest something", "just drinks please", or "bring the soup, no matter what".

An agent that acts by asking for one named tool with structured arguments at a time.

Each action is a request like get_sales with a store and a period, which your code checks and runs. It is simpler to audit and safer than a code agent, because the model can only choose from your menu of tools. It can need more steps for loops and calculations.

Like ordering from a fixed menu: you can only ask for what is listed, one dish at a time.

A ready-made LangGraph step that runs whatever tools the model just asked for.

ToolNode(tools) reads the tool calls on the last AI message, runs each tool, and returns one ToolMessage per call, catching errors along the way. It pairs with tools_condition, a router that sends the graph to the node named "tools" when the model asked for tools and to END otherwise. Together they give you the agent loop in a few lines.

Like a kitchen runner who fetches exactly what the chef shouted for and pins each result back on the rail.

The k highest-scoring results, for example the top 5 matches for a search.

Search and retrieval systems score many candidates and return only the best k. Choosing k is a trade-off: too small and you miss good answers, too large and you include noise. In RAG systems, k decides how many retrieved chunks are handed to the language model.

Like a shop assistant who shows you the five most suitable cakes rather than every cake in the store.

A setting that limits the choice of next token to the smallest group of top tokens whose probabilities add up to p.

With top_p = 0.9, the model keeps the most likely tokens until their probabilities reach 90%, discards the rest, and samples from the survivors. It adapts to the situation: when the model is confident the group is tiny, and when it is unsure the group is larger. Change it or temperature, usually not both.

Like trimming a long menu to the dishes that make up 90% of orders before choosing.

A detailed record of everything one request did, step by step, including every model call and its timing.

A trace is usually shown as a tree: the overall run at the top, with nodes, model calls and tool calls nested underneath, each with inputs, outputs, token counts and durations. Tracing tools such as LangSmith or Langfuse collect them automatically. Traces contain the full text users sent, so they must be handled as personal data.

Like a parcel tracking history that lists every depot the parcel passed through and how long it waited at each.

The error report Python prints when a program crashes, showing what went wrong and where.

A traceback lists the chain of function calls that led to the error, with file names and line numbers, and ends with the error type and message. Read it from the bottom up: the last line names the problem, and the lines above show where it happened.

Like a delivery tracking history read backwards: the last entry tells you where the parcel got stuck.

Dividing labelled data into one part to learn from and a separate part, kept hidden, to measure with.

A model is trained on the training set and scored on the test set, which it has never seen, so the score reflects how it will do on new data. A common split is 80/20, with stratification to keep the same mix of labels in both parts. Duplicates that land on both sides make the test too easy.

Like practising with past exam papers but being marked on a paper you have never seen.

The process of setting a model's parameters by showing it many examples and correcting its mistakes.

Training repeats a loop: predict, measure how wrong the predictions are (the loss), nudge every parameter to reduce that error, and repeat. For a two-knob model it takes seconds; for a large language model it takes weeks on thousands of GPUs and is done by the provider, not by you.

Like years of apprenticeship: slow, expensive, and finished before the baker serves a single customer.

The repeated cycle of predict, measure the loss, compute gradients and update parameters.

In PyTorch the loop runs five steps per mini-batch: forward pass, loss, optimizer.zero_grad(), loss.backward(), optimizer.step(). Around it sit epochs, validation checks, logging and early stopping. Keras hides the loop inside model.fit, but the same steps happen.

Like a baker practising: bake a tray, taste, compare with the recipe, adjust, bake again.

A group of database changes that are saved all together, or not at all.

If any change in a transaction fails, the whole group is rolled back, so the data never ends up half-written. In Python's sqlite3, `with conn:` wraps the block in a transaction that commits on success and rolls back on error.

Like a bank transfer: the money leaves one account and arrives in the other, or neither happens.

Starting from a model already trained on a large dataset and adapting it to your task.

A network pretrained on millions of images has already learned edges, textures and shapes. Replacing its final layer and training on a few hundred of your own photos often beats training from scratch on the same data. The same idea underlies fine-tuning LLMs.

Like hiring an experienced pastry chef and teaching them your menu, instead of training someone who has never baked.

The neural network design that almost every modern language model is built on.

Introduced in the 2017 paper "Attention Is All You Need", the transformer processes all the words of a text in parallel and uses attention to relate them to each other. It scales well to huge datasets and GPUs, which is why GPT, Claude, Gemini and Llama all use it. You build one in Module 30.

Like a standard floor plan that most modern bakeries copy because it works so well.

A reasoning method where a model proposes several possible next steps, evaluates them, and explores the most promising, backtracking from dead ends.

It turns reasoning into a search over partial solutions. It can solve puzzles a single chain of thought misses, at the cost of many more model calls. If a program can check or search the answer exactly, a tool call is usually cheaper.

Like a chess player considering several moves and dropping the lines that clearly lose.

The smallest graph fact: a subject, a relationship and an object, like (Aarav, ALLERGIC_TO, peanuts).

Any graph can be written as a list of triples, and LLMs are often asked to extract triples from text to build graph memory. Raw triples need their names resolved to canonical nodes before they are stored, or the graph fills with duplicates.

Like one card, one piece of string and the card at the other end.

Of the items that really are negative, the share a checker correctly labels negative.

When aligning an LLM judge with "pass" as positive, TNR is the share of human-failed outputs that the judge also fails. Because failures are usually rare, a judge can have high accuracy and a terrible TNR, which means it misses the problems you care about most.

Like the share of bad cakes an inspector actually catches.

Of the items that really are positive, the share a checker correctly labels positive.

When aligning an LLM judge with "pass" as positive, TPR is the share of human-passed outputs that the judge also passes. Some write-ups treat "fail" as positive instead, which swaps the names. Always report it alongside the true negative rate.

Like the share of good cakes an inspector correctly lets through.

How Python decides whether a value counts as true or false in an `if`.

`None`, `False`, `0`, `0.0`, empty strings and empty collections are falsy; almost everything else is truthy. This is why `value or default` replaces a real `0` with the default. Use `is None` when zero or empty is a valid answer.

Like a bouncer who turns away anyone with an empty hand, including the person holding a valid ₹0 voucher.

Python's way of running risky code and deciding what to do if it fails.

Code in `try` runs normally; if a matching exception is raised, the `except` block handles it. `else` runs only on success and `finally` always runs. Catch only the specific exceptions you can do something useful about.

Like a backup plan written next to a recipe step: "if the cream splits, do this".

Deciding when the person speaking has finished, so the agent can reply.

Silence-based turn detection waits for a set amount of silence, often 300 to 800 ms; semantic turn detection also checks whether the words sound finished. It is often the largest single item in a voice agent's latency budget. Too short a wait interrupts hesitant callers; too long feels sluggish.

Like a good waiter who can tell the difference between a customer pausing to think and a customer who has finished ordering.

A tool that reads your code without running it and reports type mistakes.

mypy and Pyright compare your type hints with how values are actually used, and flag mismatches before the code runs. They cannot see data that only exists at runtime, such as an API or LLM reply. For that, use runtime validation.

Like a food-safety inspector who checks every jar label against its contents before service.

A label in code that says what type a variable, parameter or return value should be.

Written like `price: int` or `def total(...) -> int`. Python ignores type hints when running code; they are for people, editors and type checkers. Hints like `str | None`, `list[str]` and `Literal[...]` describe more complex shapes.

Like the label on a pantry jar: it says "sugar", but it cannot stop someone pouring in salt.

A method for drawing high-dimensional data in 2-D so that close neighbours stay close.

UMAP finds each point’s nearest neighbours in the full space, then arranges the points on a flat page so those neighbours stay together. It shows clusters much more clearly than PCA. Distances between clusters and cluster sizes on a UMAP plot are not reliable, and the picture changes with the random seed.

Like peeling an orange and pressing the peel flat: nearby bits stay nearby, but the overall shape tears and stretches.

Someone making your AI app use huge amounts of compute, running up your bill or slowing it down.

Attackers can send floods of long requests, trigger expensive agent loops or query your model at scale to copy its behaviour. Defences include per-user rate limits, maximum input sizes, max_tokens on every call, step limits on agents, spending alerts and hard budget caps.

Like a prank caller ordering a hundred cakes a minute on a line where every call costs you money.

What one unit of work, such as one resolved conversation, costs compared with the value it creates.

For an AI feature, the useful unit is usually a resolved task or conversation, not a single API call. Comparing that cost with the alternative, such as a person handling the same task, tells you whether the feature is worth running and where to optimise. Exchange rates and price changes move it, so recheck it regularly.

Like a bakery knowing what each croissant costs to make before deciding whether to sell it at ₹45.

A small automated check that one piece of code behaves as expected.

Unit tests call a function with known inputs and compare the output with the expected result. They are the best judge of generated code, and tests you have read act as a specification the model cannot quietly reinterpret. They only check what they cover.

Like tasting each batch of icing before it goes on the cakes.

An open-source library that makes fine-tuning open models faster and lighter on memory.

It wraps Hugging Face's Transformers, PEFT and TRL with optimised kernels, and adds helpers for loading 4-bit models, applying chat templates, training only on responses and exporting to GGUF for Ollama. Its API changes often, so its maintained notebooks are the best reference.

Like a well-organised prep station that lets one cook do the work of three.

Learning patterns from data that has no answers attached.

Instead of learning input-to-label mappings, the model learns the structure of the data itself, for example by reconstructing it (autoencoders) or predicting hidden parts of it. Large language models are pretrained this way on raw text before being fine-tuned. It matters because unlabelled data is plentiful and labels are expensive.

Like sorting a box of buttons into groups without anyone telling you what the groups should be.

Insert a record, or replace it if a record with the same ID already exists.

Upsert combines update and insert. With stable IDs, re-running a loading script upserts the same records instead of creating duplicates, and edited documents replace their old versions. If a document now produces fewer chunks, the extra old ones must still be deleted separately.

Like a filing clerk who swaps in the new copy of a folder if it is already there, and files it fresh if it is not.

The standard way computers store text as bytes, using one byte for plain English letters and up to four for other characters.

A Devanagari letter takes three bytes and most emoji take four. Byte-level tokenizers start from UTF-8 bytes, which is why text in some scripts begins at a disadvantage in token count. A single byte from the middle of a character is not valid text on its own.

Like a postal system where short local addresses fit on one line and international ones need three or four.

A fast tool that installs Python, creates project environments and manages a project's libraries.

uv replaces several older tools (pip, venv, pyenv) with one. `uv init` creates a project, `uv add` installs a library and records it in pyproject.toml and uv.lock, and `uv run` runs code inside the project's environment. This course uses it everywhere.

Like a kitchen manager who stocks each station with exactly the ingredients its recipe needs, and writes the list down.

An autoencoder that encodes each input as a fuzzy cloud of codes and keeps all the clouds near the centre, so random codes decode into sensible new outputs.

Its loss has two parts: a reconstruction term and a KL term that pulls each cloud towards a standard bell curve. It trains stably and gives a smooth latent space, at the cost of somewhat blurry images. VAE encoders and decoders are used inside latent diffusion models such as Stable Diffusion.

Like a city map where every neighbourhood overlaps its neighbours, so wherever you drop a pin, there is a house.

Checking that data has the right shape and acceptable values before using it.

Validation asks whether a value is allowed, such as a quantity between 1 and 50. Coercion converts a value to the right type when it is safe. Validate at the boundaries of your app, wherever data arrives from users, APIs or models.

Like checking a cake order slip is complete and sensible before the baker starts.

Data held back from training and used to make decisions such as when to stop or which model to keep.

Data is usually split three ways: training (to learn), validation (for choices during development) and test (touched once, for an honest final score). Using the test set to make decisions makes its score optimistic. For time-based data, split by date.

Like mock exams before the real board exam, which you only sit once.

An open-source fork of Redis, run by the Linux Foundation, that works with the same clients and commands.

Valkey was created in 2024 after Redis changed its licence, continuing from the last open-source Redis release. It speaks the same protocol, so redis-py, RQ and the tools in this module work with it unchanged. Use whichever your organisation already runs.

Like a recipe passed to a new bakery that keeps making the same bread under a new name.

When the learning signal shrinks to almost nothing as it passes back through many layers or time steps.

Backpropagation multiplies slopes layer by layer; if each is below 1, the product shrinks exponentially, so early layers barely learn. Sigmoid and tanh activations, poor initialisation and long sequences make it worse. ReLU-family activations, good initialisation, normalisation layers, residual connections and LSTM gates all fight it.

Like a whispered message passed down a long line of people, fading until the first person hears nothing.

A name you attach to a value so you can use it later.

In Python, `price = 85` sticks the name `price` on the value 85. Assigning again moves the name to a new value. A variable remembers the value it holds, not how that value was calculated.

Like the price tag on a bakery tray: change the tag once and everyone who reads it sees the new price.

An ordered list of numbers, such as [0.12, -0.80, 0.33].

In AI work, vectors usually represent things like text or images as points in a space with many dimensions. An embedding is a vector. Comparing vectors, for example by the angle between them, tells you how similar the things they represent are.

Like a recipe written only as quantities in a fixed order: flour, sugar, butter, eggs.

A database built to store huge numbers of vectors and quickly find the ones closest to a given vector.

Vector databases such as Chroma, Pinecone, Milvus or Postgres with pgvector power semantic search and RAG. They use special indexes to find near neighbours among millions of vectors in milliseconds. Module 08 covers how they work and how to choose one.

Like a librarian who shelves books by topic so well she can hand you the five most related books in a second.

Doing an operation on a whole array at once instead of looping over its items.

`prices * qty` multiplies a million pairs in one step, often tens of times faster than a Python loop. Training and running neural networks depend on it, which is why GPUs, built for many simple sums at once, matter so much.

Like washing a whole tray of eggs under one tap instead of one egg at a time.

The LlamaIndex index that embeds every node and stores the vectors so the most similar ones can be found quickly.

It is the index used for most RAG work in LlamaIndex. By default it keeps vectors in memory; you can persist it to disk or back it with a vector database such as Chroma, Pinecone or pgvector. Other index types exist for special jobs, such as summarising every node.

Like a card catalogue where cards about similar topics are filed next to each other.

Depending on one supplier so heavily that switching becomes slow or expensive.

With LLMs, lock-in lives less in the SDK calls than in tuned prompts, evaluation results and provider-specific features. A thin adapter of your own makes switching possible; testing on the alternative makes it safe.

Like a bakery whose recipes only work with one supplier’s flour.

Handling more load by making one machine or component more powerful.

A bigger server, more memory or a faster model are vertical scaling. It is simple but has a ceiling and leaves a single point of failure. Real systems often combine both: reasonably sized machines, several of them.

Like buying a bigger, faster oven instead of hiring more bakers.

A model that understands or generates video, treating it as a sequence of frames over time.

Video understanding models classify clips or answer questions about them, sometimes by sampling frames for an image model. Video generation models extend diffusion to many frames at once so that motion stays consistent. Generation needs large GPUs or hosted services and minutes per short clip, making video the most expensive modality.

Like a flip-book: many drawings that must all agree with each other to look like smooth motion.

A private folder of Python and libraries for one project, separate from every other project.

Separate environments stop one project's library versions from breaking another's. uv creates one automatically in a hidden `.venv` folder the first time you run or add something. Never commit it to git; it can be rebuilt exactly from uv.lock with `uv sync`.

Like giving each recipe its own labelled pantry shelf, so the Christmas cake's ingredients never get mixed into the croissants.

A transformer that reads an image by cutting it into small patches and treating each patch as a token.

A 224 x 224 image becomes 196 patches of 16 x 16 pixels, each projected to an embedding, with a [CLS] token and position embeddings added. It needs large-scale pretraining to beat convolutional networks, then fine-tunes well on small datasets.

Like reading a photo as a sentence of tiles instead of a sentence of words.

A language model that can also look at images and answer questions about them in words.

A vision-language model joins an image encoder to a language model through a small projector that turns image features into image tokens. The language model reads those tokens alongside the text prompt and generates an answer. VLMs are flexible and fluent, but they can invent details, especially small text, numbers and counts.

Like a colleague you can show a photo to and ask anything, who answers confidently even when the photo is blurry.

Answering a free-form question about an image, such as "Is the box sealed?".

Early VQA models chose answers from a fixed list; modern ones use vision-language models that generate the answer. Asking narrow, checkable questions, such as yes or no confirmations, makes answers far more reliable than open-ended reading of details.

Like asking a colleague "Is the box sealed?" while showing them the photo on your phone.

The fixed list of all the tokens a tokenizer and its model know.

Modern LLM vocabularies hold roughly 100,000 to 200,000 tokens: common words, word pieces, punctuation, and raw bytes as a fallback. The vocabulary is fixed when the tokenizer is built and never changes for that model. Each model family has its own, which is why the same text gives different token counts on different models.

Like the bakery's full stock-code list: everything you can order has a code on it.

A small model that labels each slice of audio as speech or not speech.

A VAD such as Silero processes audio in short slices and reports when speech starts and stops. Voice agents use it to decide when a caller has finished a turn and to notice when the caller starts talking over the agent. Its silence threshold trades snappy replies against cutting people off mid-sentence.

Like a light on a radio studio desk that turns on whenever someone is actually talking.

Storage that lives outside any container, so data survives when containers are replaced.

A volume is mounted into a container at a folder path, such as -v redis-data:/data. Delete the container, start a new one with the same volume, and the data is still there. Every database or model store in Docker needs one.

Like a walk-in fridge shared between shifts: the ovens change, the stock stays.

A tool that lets a model search the internet and read current pages instead of relying only on what it learned in training.

It can be a server tool run by the model provider, with built-in citations and a per-search fee, or a client tool you run with a search library or API. Results can be stale, wrong or malicious, so show sources with dates and keep web-reading agents away from powerful actions.

Like giving a researcher a phone to call around town: useful answers, but not every caller is reliable.

An HTTP request that another service sends to your server when something happens, such as a new WhatsApp message.

Instead of you asking a service for updates, it calls a URL you provide. Providers usually expect a quick acknowledgement and retry if they do not get one, so slow webhook handlers can cause duplicate deliveries. Acknowledge fast and do slow work on a queue.

Like a doorbell: the visitor tells you they have arrived, rather than you checking the door every minute.

A network connection that stays open so both sides can send messages to each other at any time.

Normal web requests are one question and one answer. A WebSocket upgrades a web connection into a long-lived, two-way channel, which suits streaming audio and events in voice agents. For browsers and phones over unreliable networks, WebRTC is often used instead because it is built for real-time media.

Like a walkie-talkie channel left on, rather than posting letters back and forth.

A small penalty that pulls every weight towards zero on each step, discouraging memorisation.

Large, finely balanced weights are often a sign of memorising individual examples. Weight decay makes big weights pay for themselves across many examples. In PyTorch it is the weight_decay argument of an optimiser, applied most cleanly by AdamW.

Like a small tax on every weight: keep it big only if it earns its keep.

How a network’s weights are set before training starts.

Weights that are too small make signals and gradients shrink layer by layer; too large and they grow. Schemes such as Kaiming (He) initialisation for ReLU and Xavier (Glorot) for tanh scale the random starting weights by the number of inputs so signals keep a steady size. For deep networks, the right initialisation can decide whether training works at all.

Like setting every amplifier in a chain to the right volume before the concert, so the sound neither fades nor screeches.

The numbers inside a layer that training adjusts: multipliers for each input, plus a starting offset.

Weights say how strongly each input affects each output; the bias shifts each output up or down regardless of input. Together they are the layer’s parameters. They start random and are learned by gradient descent.

Like the volume knobs on a mixing desk, plus a master offset for each channel.

A family of open speech recognition models from OpenAI that transcribe and translate speech in about 100 languages.

Whisper turns audio into a log-Mel spectrogram, encodes it with a transformer, and decodes the transcript token by token, with special tokens that choose the language and whether to transcribe or translate into English. Sizes range from tiny to large. Because the decoder is a text generator, Whisper can invent text when given silence or noise.

Like a multilingual stenographer who can also hand you an English version, but who doodles words when the room goes quiet.

A separate program that repeatedly takes jobs from a queue, runs them and records the results.

Each RQ worker runs one job at a time, so the number of workers sets how many jobs run in parallel and caps how many LLM calls are in flight. Workers can run on any machine that can reach the queue, which is what makes horizontal scaling straightforward.

Like a cook who takes the next order slip from the rail, cooks it and puts the plate out.

A system where a language model is called at fixed points along a path that your code decides in advance.

In a workflow, the order of steps is written in your code: classify, then draft, then check. The model does the thinking inside each step but never chooses what happens next. Workflows are cheaper, faster and easier to test than agents, and most reliable LLM products are built from them.

Like a recipe card: every step is written down, so two trainees following it make the same cake.

The folder your terminal is currently "standing in", which commands run from.

Relative paths like `notes.txt` are looked up in the working directory, so the same command can work in one window and fail in another. `pwd` prints it and `cd` changes it. Many "file not found" errors are really "you are in the wrong folder".

Like the room an assistant is standing in: "fetch the butter from the second shelf" depends entirely on which room that is.

The rules for what an agent is allowed to save into long-term memory.

A write policy decides what is worth keeping (allergies, preferences, important events) and what must never be stored (payment details, secrets, the assistant's own claims, unverified promises). A robust pattern is that the LLM proposes candidate memories in a strict format and plain code accepts or rejects each one.

Like a doctor deciding what goes into a patient's file: the diagnosis, not the chat about the weather.

Sorting items into categories without any labelled examples, using only descriptions of the categories.

With embeddings, you write a sentence describing each label, embed the descriptions, and assign each new item to the most similar description. With LLMs, you describe the labels in a prompt. It is rougher than a trained classifier but works on day one, and corrected results become labelled examples for later.

Like a new post-room worker sorting letters using only a sign on each pigeonhole that describes what goes there.

Asking a model to do a task with instructions only, and no worked examples.

Zero-shot is the simplest and cheapest approach and works well for clear-cut tasks. It struggles when your labels have private meanings or there are tricky boundary cases. Always measure it first as a baseline.

Like handing a new colleague a list of categories and walking away.