Project: RAG chatbot over YouTube transcripts
Ship a Pinecone-backed chatbot that answers questions about YouTube videos using only what was said, with clickable citations that open each video at the right second.
Words in this lessonRetrieval-augmented generation · Chunking · Pinecone · Grounding · Citation · Query rewriting
- Retrieval-augmented generation
- Finding relevant passages from your own documents first, then giving them to the model so it answers from real facts. Like an open-book exam: the student still writes the answer, but from the textbook in front of them rather than from memory.
- Chunking
- Cutting long documents into smaller passages so the relevant piece can be found and sent to a model. Like slicing a cake so you can serve one piece without carrying the whole cake to the table.
- Pinecone
- A fully managed vector database that you use over the internet with an API key. Like taking a taxi instead of owning a car: someone else handles the maintenance, and you pay per ride.
- Grounding
- Giving the model the real facts it needs inside the request, so it answers from them rather than from memory. Like handing the new hire the order sheet before they answer the customer.
- Citation
- A pointer from a claim in an answer to the exact source passage it came from. Like a footnote in a textbook that tells you which page of which book a fact came from.
- Query rewriting
- Turning a user’s message into a better search query before retrieving, for example by adding context from the conversation. Like a librarian who hears "and the sequel?" and goes looking for the sequel to the specific book you were discussing.
On this page
- The brief
- What you will build
- Architecture
- Tech stack
- The numbers for Arjun's playlist
- Build it, milestone by milestone
- 1. Fetch transcripts for your videos
- 2. Chunk transcripts into timestamped windows
- 3. Embed chunks and upsert them into Pinecone
- 4. Retrieve the most relevant timestamps
- 5. Generate grounded answers with timestamp citations
- 6. Add a chat loop with follow-up questions
- 7. Evaluate with a small question set
- Done means done
- Stretch goals
- Common pitfalls
- Show it off
Arjun has a YouTube playlist called Watch later (seriously). It holds 40 conference talks about databases, search and AI engineering, each between 30 and 60 minutes long. He added most of them a year ago. He has watched three.
The Pune developer meetup’s weekend hackathon has the theme “learning from video”. Arjun signs up on Thursday night with a one-line idea: ask my playlist a question, get an answer with links to the exact moments in the videos where it was said.
On Friday, Priya agrees to review his work. She sets one rule. “Every claim the bot makes must link to the second in the video where someone actually said it. If it can’t point to the moment, it should say it doesn’t know. A chatbot that confidently summarises a talk the speaker never gave is worse than no chatbot.”
Meera hears about it at lunch and asks the question Arjun was secretly hoping for. “If this works, could it do the same for our baking-class videos? Customers keep asking at which point in the video Chef Devika adds the butter.”
Important
What you will ship: a chatbot that answers questions about a set of YouTube videos using only what was said in them, retrieving the relevant moments from a Pinecone index of timestamped transcript chunks and citing each claim with a clickable link that opens the video at that second.
The brief
The bot takes a playlist of videos and a question in plain English, and returns:
- a short answer, written only from what the speakers said,
- numbered citations after each claim, like [1], and
- a list of sources, where each one is a link such as “Scaling search in practice, 12:34” that opens YouTube at that exact second.
If the videos do not cover the question, the bot says so plainly and does not invent an answer.
This is retrieval-augmented generation, or RAG: instead of asking a language model to answer from memory, you first retrieve relevant passages from your own data, then augment the prompt with them, then let the model generate an answer grounded in them. You will study RAG in depth in the next module. This project builds a complete, working one, using everything from this module: embeddings, chunking, a vector database, metadata, namespaces and evaluation.
You can use any videos that have captions: a lecture series, a podcast, a cooking channel. Talks with one clear speaker and good audio work best for a first build.
What you will build
Must have
- A script that fetches transcripts, with their timestamps, for a list of video IDs.
- A chunker that groups transcript lines into overlapping windows of about a minute, keeping each window’s start and end time.
- A Pinecone index holding one record per chunk, with the video ID, title, start time and text in metadata.
- A retriever that finds the most relevant moments for a question, optionally within one video.
- An answer step that uses Claude to answer only from the retrieved chunks, with numbered citations turned into timestamp links.
- A chat loop that handles follow-up questions such as “what did he say after that?”.
- An evaluation script with at least 15 questions, measuring retrieval and the “I don’t know” behaviour.
Nice to have
- A Streamlit chat interface with the video embedded and jumping to the cited time.
- Hybrid search that also matches exact words, such as product names.
- Per-video summaries generated from the chunks and stored for fast overview questions.
Architecture
flowchart LR
V["Video IDs"] --> T["Fetch transcripts"]
T --> C["Chunk by time"]
C --> E["Embed (OpenAI)"]
E --> P[("Pinecone index")]
Q["User question"] --> R["Rewrite follow-ups"]
R --> S["Retrieve top moments"]
P --> S
S --> A["Answer with Claude"]
A --> O["Answer with timestamp links"]
Fetch transcripts. Downloads the caption track for each video as a list of short snippets: a few words each, with a start time and duration in seconds. Saved to disk as JSON so you never fetch the same video twice.
Chunk by time. One snippet (“so the first thing”) is too short to mean anything. A whole 45-minute talk is far too long for one vector. This step groups snippets into windows of about 60 seconds, overlapping by 15 seconds, and remembers when each window starts.
Embed. Turns each chunk’s text into a vector with OpenAI’s text-embedding-3-small (1,536 numbers).
Pinecone index. Stores every chunk’s vector with metadata: video ID, title, start and end time, and the text itself. One namespace per playlist keeps separate collections of videos apart.
Rewrite follow-ups. A follow-up like “and what about its downsides?” means nothing on its own. This step uses the conversation so far to turn it into a standalone question before searching.
Retrieve top moments. Embeds the question, queries Pinecone, removes overlapping duplicates, and keeps the best few chunks.
Answer with Claude. Sends the question and the retrieved chunks, numbered, with strict instructions: answer only from these, cite each claim, say so if the answer is not there.
Answer with timestamp links. Turns each [n] citation into a link that opens the video at the chunk’s start time.
Tech stack
| Piece | Tool | Why |
|---|---|---|
| Transcripts | youtube-transcript-api |
Fetches existing captions with timestamps, no API key needed |
| Embeddings | OpenAI text-embedding-3-small |
Strong quality, 1,536 dimensions, very cheap |
| Vector database | Pinecone (serverless) | Managed, metadata filters, namespaces per playlist |
| Answers | Claude (claude-opus-5-5) via the Anthropic SDK |
Follows grounding and citation instructions well |
| Secrets | python-dotenv |
Keys stay out of code |
| Interface | Terminal first, Streamlit as a stretch | Get the logic right before the looks |
uv init yt-rag && cd yt-rag
uv add youtube-transcript-api openai pinecone anthropic python-dotenv
Your .env needs OPENAI_API_KEY, PINECONE_API_KEY and ANTHROPIC_API_KEY.
The numbers for Arjun’s playlist
Work these out before building, so nothing surprises you:
- Talk time: 40 talks × about 45 minutes = 30 hours.
- Words: speakers average roughly 150 words a minute, so about 270,000 words, roughly 360,000 tokens.
- Chunks: 60-second windows that advance by 45 seconds give about 60 chunks per 45-minute talk, so about 2,400 chunks.
- Embedding cost: with overlap, about 480,000 tokens. At about 0.02 US dollars per million tokens, roughly 0.01 dollars, under ₹1, once.
- Pinecone storage: 2,400 records × about 7 KB (vector plus metadata text) ≈ 17 MB. Well within the free starter tier at the time of writing.
- Per question: one embedding (negligible), one Pinecone query, and one Claude call with about 1,500 input tokens (instructions plus five chunks) and up to 400 output tokens. Multiply by the current price on Anthropic’s pricing page to get your cost per question, and log it.
Build it, milestone by milestone
1. Fetch transcripts for your videos
Why. Everything downstream depends on having the text and the time each line was spoken. Without timestamps, you can answer, but you cannot point to the moment, which is Priya’s rule.
List your videos in videos.json. The video ID is the part after v= in a YouTube URL. Titles are written by hand here, which keeps the transcript library’s job small.
[
{"id": "abc123XYZ00", "title": "Scaling search in practice"},
{"id": "def456UVW11", "title": "Postgres as a vector database"}
]
# fetch.py
import json
from pathlib import Path
from youtube_transcript_api import YouTubeTranscriptApi
OUT = Path("data/transcripts")
OUT.mkdir(parents=True, exist_ok=True)
ytt = YouTubeTranscriptApi()
for video in json.load(open("videos.json")):
path = OUT / f"{video['id']}.json"
if path.exists():
continue # never fetch twice
try:
fetched = ytt.fetch(video["id"], languages=["en"])
snippets = fetched.to_raw_data() # [{"text", "start", "duration"}, ...]
except Exception as err: # no captions, private video, blocked...
print(f"skipped {video['id']}: {type(err).__name__}")
continue
path.write_text(json.dumps({**video, "snippets": snippets}))
print(f"{video['title']}: {len(snippets)} snippets")
This library reads captions that already exist on YouTube; it does not transcribe audio. Library APIs change quickly, and this one has changed its main function names before: older versions used YouTubeTranscriptApi.get_transcript(video_id) instead of creating an object and calling fetch. If a name has moved, check the project’s page on PyPI or GitHub.
What working looks like. One JSON file per video in data/transcripts/, and a line per video such as Scaling search in practice: 1184 snippets. Each snippet looks like {"text": "so the first thing we tried", "start": 754.2, "duration": 2.9}.
Most likely bug. Some videos are skipped. Many have captions disabled, or only in another language, and YouTube sometimes blocks requests from cloud servers. Run the fetch from your own machine, try languages=["en", "en-US"], and replace videos that have no captions.
Caution
Transcripts belong to the people who made the videos. Use them for your own learning and search, keep the bot’s answers short and linked back to the original video, and do not republish whole transcripts. Check YouTube’s terms and the creator’s wishes before building anything public on someone else’s content.
Stuck? Hint
To see which languages a video offers, the same library can list available transcripts for a video ID; the GitHub README shows how. Auto-generated English captions are usually available even when the creator did not upload any, and they are good enough for retrieval, though names and technical terms are often misspelled.
2. Chunk transcripts into timestamped windows
Why. A snippet is 2 to 5 seconds of speech, too little to carry an idea. A whole video is far too much for one vector: its embedding would point at the average of every topic in the talk. A minute of speech usually holds one idea. Overlap makes sure an idea that straddles a boundary appears whole in at least one chunk.
# chunk.py
def chunk_transcript(snippets, window=60.0, overlap=15.0):
"""Group snippets into windows of about `window` seconds, overlapping by `overlap`."""
chunks, i = [], 0
while i < len(snippets):
t0 = snippets[i]["start"]
j = i
while j < len(snippets) and snippets[j]["start"] < t0 + window:
j += 1 # take snippets that start inside the window
piece = snippets[i:j]
chunks.append({
"start": t0,
"end": piece[-1]["start"] + piece[-1]["duration"],
"text": " ".join(s["text"].replace("\n", " ") for s in piece),
})
if j >= len(snippets):
break
next_start = t0 + window - overlap # step forward 45 seconds
while i < len(snippets) and snippets[i]["start"] < next_start:
i += 1
return chunks
Trace one window from a talk. Snippets starting at 754.2 s (“so the first thing we tried”), 757.1 s (“was chunking by a fixed number of tokens”), and onwards up to 812.9 s are collected into one chunk: start 754.2, end about 815.6, about 150 words. The next chunk starts at the first snippet after 799.2 s (754.2 + 45), so the final 15 seconds of this chunk appear again at the start of the next.
What working looks like. A 45-minute talk produces about 60 chunks. Every chunk’s start is earlier than its end, consecutive chunks overlap, and printing three random chunks gives readable paragraphs of speech.
Most likely bug. Overlap larger than the window, or equal to it. The step forward becomes zero and the loop never moves. Keep overlap well below window.
Stuck? Hint
Write a tiny test with five fake snippets at 0, 20, 40, 60 and 80 seconds, each 20 seconds long, and window=50, overlap=10. You should get two chunks: the first covering the snippets at 0, 20 and 40, and the second starting at 40 and covering 40, 60 and 80. The snippet at 40 appears in both: that is the overlap. If you get something else, print i and j each time round the loop.
3. Embed chunks and upsert them into Pinecone
Why. Searching 2,400 chunks by meaning needs their vectors in a database that can filter by video and return the timestamps with each match. Pinecone stores the metadata right next to the vector, so a match comes back ready to cite.
# index.py
import json
import os
import time
from pathlib import Path
from dotenv import load_dotenv
from openai import OpenAI
from pinecone import Pinecone, ServerlessSpec
from chunk import chunk_transcript
load_dotenv()
oai = OpenAI()
pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"])
INDEX, NAMESPACE = "yt-rag", "conf-talks"
if INDEX not in pc.list_indexes().names():
pc.create_index(name=INDEX, dimension=1536, metric="cosine",
spec=ServerlessSpec(cloud="aws", region="us-east-1"))
while not pc.describe_index(INDEX).status["ready"]:
time.sleep(1)
index = pc.Index(INDEX)
def embed(texts):
resp = oai.embeddings.create(model="text-embedding-3-small", input=texts)
return [d.embedding for d in resp.data]
for path in Path("data/transcripts").glob("*.json"):
video = json.loads(path.read_text())
chunks = chunk_transcript(video["snippets"])
for b in range(0, len(chunks), 100): # batches of 100
batch = chunks[b:b + 100]
vectors = embed([c["text"] for c in batch])
index.upsert(namespace=NAMESPACE, vectors=[{
"id": f"{video['id']}#{int(c['start'])}", # stable: same chunk, same ID
"values": v,
"metadata": {"video_id": video["id"], "title": video["title"],
"start": c["start"], "end": c["end"], "text": c["text"]},
} for c, v in zip(batch, vectors)])
print(f"{video['title']}: {len(chunks)} chunks")
print(index.describe_index_stats())
What working looks like. One line per video, then stats showing your namespace with about 2,400 records for 40 talks. Re-running the script does not increase the count, because the IDs are stable and upsert replaces.
Most likely bug. A dimension error, because the index was created earlier with a different dimension. Delete the index in the Pinecone console and let the script recreate it, or use a new index name.
Stuck? Hint
If the count is lower than expected right after loading, wait a few seconds and check again: serverless indexes are eventually consistent. If it is higher than expected after a re-run, your IDs are not stable, probably because you used a random ID or the chunk index instead of the video ID and start time.
4. Retrieve the most relevant timestamps
Why. The answer can only be as good as the moments you hand to the model. Retrieval must find the right moments, avoid near-duplicates from overlapping chunks, and optionally stay within one video.
Follow one question through it: “What did the speakers say about choosing a chunk size?”
- The question is embedded into 1,536 numbers.
- Pinecone returns the 8 nearest chunks in the
conf-talksnamespace, each with a score. The top one: Scaling search in practice, start 754.2 s, score 0.71. - Two of the 8 are overlapping windows from the same talk (754 s and 799 s). They say much the same thing, so only the higher-scoring one is kept.
- The best 5 remaining become the context for the answer.
# retrieve.py
def retrieve(question, k=5, video_id=None, min_score=0.30):
q = embed([question])[0]
res = index.query(vector=q, top_k=k * 2, include_metadata=True, namespace=NAMESPACE,
filter={"video_id": {"$eq": video_id}} if video_id else None)
picked = []
for m in res["matches"]:
if m["score"] < min_score:
break # sorted best first
md = m["metadata"]
overlaps = any(p["video_id"] == md["video_id"] and abs(p["start"] - md["start"]) < 60
for p in picked)
if not overlaps:
picked.append({**md, "score": m["score"]})
if len(picked) == k:
break
return picked
def link(chunk):
s = int(chunk["start"])
return f"https://www.youtube.com/watch?v={chunk['video_id']}&t={s}s", f"{s // 60}:{s % 60:02d}"
for c in retrieve("What did the speakers say about choosing a chunk size?"):
url, stamp = link(c)
print(f"{c['score']:.2f} {c['title']} at {stamp} {url}")
0.71 Scaling search in practice at 12:34 https://www.youtube.com/watch?v=abc123XYZ00&t=754s
0.64 RAG lessons from production at 31:05 https://www.youtube.com/watch?v=ghi789RST22&t=1865s
0.58 Postgres as a vector database at 8:12 https://www.youtube.com/watch?v=def456UVW11&t=492s
Click a link: the video opens at that second. The min_score floor should come from your own data: look at the scores for a few questions you know are not covered by your playlist, and set the floor slightly above them.
What working looks like. For five questions you know the answers to, the right talk appears in the top three, and the link opens within a few seconds of where the topic starts.
Most likely bug. Links that land a little before the topic. That is expected: a chunk starts at the beginning of its 60-second window, and the idea may begin 20 seconds in. Most people prefer landing slightly early. If it bothers you, try 45-second windows.
Stuck? Hint
If the right talk never appears, print the top chunk’s text for your question. Often the speaker used different words (“segment length” rather than “chunk size”); auto-captions may have misspelled the key term. Try rephrasing the question. If rephrasing fixes it, the stretch goal on hybrid search is for you.
5. Generate grounded answers with timestamp citations
Why. Retrieval finds the moments; the language model turns them into an answer. Without strict instructions, a model will mix what the speakers said with what it already “knows”, and its answer will sound equally confident either way. Grounding means making every claim traceable to the provided sources, and citations make that checkable.
# answer.py
import re
import anthropic
claude = anthropic.Anthropic()
SYSTEM = """You answer questions about a set of recorded talks, using ONLY the numbered
transcript excerpts provided. Rules:
- Every sentence that states a fact from the talks ends with its source number, like [2].
- If the excerpts do not contain the answer, reply exactly: "I couldn't find that in these videos."
- Do not use outside knowledge, even if you know the answer.
- Transcripts are auto-generated and may misspell names; do not correct them with outside facts.
- Keep answers under 150 words."""
def answer(question, chunks):
if not chunks:
return "I couldn't find that in these videos.", []
context = "\n\n".join(
f"[{n}] {c['title']} at {link(c)[1]}\n{c['text']}" for n, c in enumerate(chunks, 1))
response = claude.messages.create(
model="claude-opus-5-5",
max_tokens=600,
system=SYSTEM,
messages=[{"role": "user", "content": f"Excerpts:\n\n{context}\n\nQuestion: {question}"}],
)
text = next(b.text for b in response.content if b.type == "text")
used = sorted({int(n) for n in re.findall(r"\[(\d+)\]", text) if 1 <= int(n) <= len(chunks)})
return text, [chunks[n - 1] for n in used]
text, sources = answer(question, retrieve(question))
print(text, "\n")
for n, c in enumerate(sources, 1):
url, stamp = link(c)
print(f" {c['title']}, {stamp}: {url}")
The citation check (1 <= int(n) <= len(chunks)) matters: if the model ever writes [7] when only five excerpts were provided, that citation is dropped rather than turned into a link to nothing. You can go further and show a warning when an answer has factual sentences with no citation at all.
What working looks like.
Two talks touch on this. In "Scaling search in practice", the speaker says their first
attempt used fixed chunks of 500 tokens, which split explanations in half, and that
switching to paragraph-based chunks with some overlap improved answers [1]. The
"RAG lessons from production" talk recommends testing several sizes on real questions
rather than picking one up front [2].
Scaling search in practice, 12:34: https://www.youtube.com/watch?v=abc123XYZ00&t=754s
RAG lessons from production, 31:05: https://www.youtube.com/watch?v=ghi789RST22&t=1865s
Click each link and check that the speaker really says it. This is the test Priya will run, so run it first.
Most likely bug. The model answers a question the videos do not cover, from its own knowledge. Ask your bot “what is the capital of France?” If it answers “Paris”, your instructions are not working.
Stuck? Hint
First check retrieval: an off-topic question should return no chunks at all if your min_score is set well, so the model never gets the chance. If chunks do come back, make the refusal instruction more prominent and keep the exact refusal wording, so you can detect it in code. Testing ten out-of-scope questions is part of milestone 7.
6. Add a chat loop with follow-up questions
Why. People ask follow-ups. “What did she say about HNSW?” is fine on its own. “And what were the downsides?” retrieves nonsense, because the word “downsides” alone matches every talk. The fix is to rewrite each follow-up into a standalone question before retrieving. This is one form of query rewriting, which you will see again in the RAG module.
REWRITE = """Rewrite the user's latest message as a standalone search question, using the
conversation for context. Reply with the question only."""
def standalone(history, message):
if not history:
return message
convo = "\n".join(f"{role}: {text}" for role, text in history[-6:]) # last 3 exchanges
response = claude.messages.create(
model="claude-opus-5-5", max_tokens=100, system=REWRITE,
messages=[{"role": "user", "content": f"{convo}\nuser: {message}"}],
output_config={"effort": "low"}, # a quick, simple task
)
return next(b.text for b in response.content if b.type == "text").strip()
history = []
while True:
message = input("\nYou: ").strip()
if message in {"", "quit", "exit"}:
break
question = standalone(history, message)
text, sources = answer(question, retrieve(question))
print(f"\n(searching for: {question})\n\nBot: {text}")
for c in sources:
url, stamp = link(c)
print(f" {c['title']}, {stamp}: {url}")
history += [("user", message), ("assistant", text)]
Printing the rewritten question is deliberate: when an answer is wrong, the first thing to check is what was actually searched for.
What working looks like. “What did the Postgres talk say about HNSW?” followed by “and the downsides?” searches for something like “downsides of HNSW indexes in the Postgres as a vector database talk”, and the answer cites that talk.
Most likely bug. The rewrite changes a perfectly good standalone question into something worse. If the user’s message is already complete, the rewrite should return it unchanged; add that line to the instructions and test with a few standalone questions.
Stuck? Hint
If you want answers about “this video” to stay within one video, keep the video ID of the most recent top source in the conversation state, and pass it as video_id to retrieve when the user says “this talk” or “this video”. Pinecone’s metadata filter does the rest.
7. Evaluate with a small question set
Why. A demo with three hand-picked questions proves nothing. Priya will ask: how often does it find the right moment, how often are its citations right, and does it refuse when it should? Twenty minutes of building an evaluation set answers all three, and lets you tell whether a change (window size, min_score, the prompt) made things better or worse.
Create eval.json with at least 15 questions you have checked by hand: 12 that the videos answer, with the video and the approximate time, and 3 that they do not.
[
{"q": "What chunk size did the first team try?", "video_id": "abc123XYZ00", "around": 760},
{"q": "Who won the 2011 cricket World Cup?", "video_id": null, "around": null}
]
# evaluate.py
import json
REFUSAL = "I couldn't find that in these videos."
cases = json.load(open("eval.json"))
hits = answered = refused_ok = 0
for case in cases:
chunks = retrieve(case["q"])
text, sources = answer(case["q"], chunks)
if case["video_id"] is None:
refused_ok += REFUSAL in text
continue
answered += 1
hits += any(c["video_id"] == case["video_id"] and abs(c["start"] - case["around"]) <= 90
for c in chunks)
print(f"retrieval hit rate: {hits}/{answered}")
print(f"correct refusals: {refused_ok}/{len(cases) - answered}")
A hit means a retrieved chunk is from the right video and starts within 90 seconds of the moment you noted. Then, by hand, open the cited links for five answers and mark each citation as supporting the claim or not. That gives you a citation accuracy number too.
What working looks like. Three numbers you can quote, for example: retrieval hit rate 11 out of 12, correct refusals 3 out of 3, citations supporting their claims 14 out of 15.
Most likely bug. A hit rate that changes every run. Pinecone results are stable for the same data, but Claude’s answers vary a little between runs. Keep retrieval scoring (which is stable) separate from answer scoring, and read the answers rather than only counting them.
Stuck? Hint
When a retrieval miss shows up, print the top three chunks for that question and the chunk you expected. If the expected chunk’s text uses different words, try a different window size or a question phrased more like the speaker. If it is not in the index at all, check whether that video’s transcript was fetched.
Done means done
- Transcripts for at least 10 videos are fetched and saved, and re-running the fetch skips them.
- Chunks are about 60 seconds long with overlap, and each has a start and end time.
- Pinecone holds one record per chunk, with stable IDs, in a named namespace; re-running the load does not change the count.
- Every timestamp link opens the right video within a few seconds of the cited moment.
- Answers use only the retrieved excerpts, cite with [n] Answers use only the retrieved excerpts, cite with **[n]**, and invalid citation numbers are dropped.
- Out-of-scope questions get the refusal message.
- Follow-up questions are rewritten, and the rewritten question is shown.
- The evaluation reports retrieval hit rate, correct refusals and hand-checked citation accuracy on at least 15 questions.
- No API key appears anywhere in the repository.
Stretch goals
- Streamlit chat app. Use
st.chat_inputandst.chat_messagefor the conversation, andst.video(url, start_time=seconds)to play the cited moment beside the answer. - Hybrid search. Speakers’ product names and acronyms are often misspelled by auto-captions, and semantic search struggles with rare exact terms. Add keyword matching alongside vector search, as you will learn in the RAG module.
- Speaker-aware chunks. For interviews and podcasts, keep speaker changes as chunk boundaries so a chunk does not mix two people’s views.
- Summaries per video. Generate a short summary for each video from its chunks, store it in a second namespace, and use it for questions like “which talks are about Postgres?”.
- Crumb’s baking classes. Point it at Crumb’s own channel and answer Meera’s question: at which point does Chef Devika add the butter?
Common pitfalls
- Whole-video or single-snippet chunks. One vector per video blurs everything together; one per snippet has too little meaning. Windows of about a minute, with overlap, are a good start.
- Losing the timestamp. If
startis not in the metadata, you cannot cite. Check one record in the Pinecone console before going further. - Trusting fluent answers. The model’s tone is equally confident when it is right and when it is making things up. Click the links.
- No refusal path. Without a minimum score and a refusal instruction, the bot answers everything, including questions the videos never touched.
- Treating transcripts as perfect. Auto-captions misspell names and terms. Expect it, and do not let the model “correct” them from outside knowledge.
- Fetching from a server. YouTube may block transcript requests from cloud machines. Fetch locally and upload the saved JSON.
Show it off
- README. The problem in two sentences, an architecture diagram, your evaluation numbers, a short note on how grounding and refusals work, and the commands to run it.
- Demo video. Ninety seconds: ask a question, show the answer with citations, click a citation and let the video play the exact moment, ask a follow-up, then ask something the videos do not cover and show the refusal.
- LinkedIn post. Lead with the experience, not the stack: “I turned a 30-hour playlist I’d never watched into a chatbot that answers with links to the exact second each thing was said. The hard part wasn’t the AI. It was making it say ‘I don’t know’ when the videos didn’t cover something. Here’s how I tested that.”
On Sunday afternoon, Arjun demos to the hackathon judges. One judge asks, “Did anyone talk about the cost of re-embedding?” The bot answers in two sentences, with one citation. The judge clicks it, and the video opens on a speaker saying almost exactly that. Then she asks who won the last football World Cup, and the bot replies, “I couldn’t find that in these videos.” She laughs and writes something on her score sheet.
Arjun does not win first place. He wins “most useful”, and a mentor from a Bengaluru startup asks for his GitHub link.
On Monday, Priya sits down with him. “You built a RAG system this weekend without calling it that. Chunking, retrieval, grounding, citations, refusals, evaluation. Next module, we take every one of those pieces apart and make it properly good: better chunking, hybrid search, reranking, and real metrics for whether the answers are faithful.” Meera, passing by, adds, “And then the baking videos.”