Bonus project: AI trading research agent

Kabir burst into the office on Friday evening with a printed poster: FinHack Pune: 36 hours, build with markets and AI. “Arjun, you’re on my team. We build an AI that tells you which stocks to buy. We’ll win. Then we’ll be rich.”

Arjun was tempted. By 11 p.m. he had a first version. It pasted the week’s headlines about a company into a prompt and asked Claude: “Based on this news, should I buy this stock? Give a target price.” The answer was confident and well written: “Buy. Strong momentum and positive sentiment. Target ₹4,500 within three months.”

On Saturday morning Priya, who had agreed to review their work, read it over coffee and asked four questions.

“Which sentence in the news supports ‘strong momentum’? At what price would this idea be wrong? Did any of those headlines come out after the date you’re pretending it is? And when someone loses money following this, who is responsible?”

Arjun had no answer to any of them. Kabir looked deflated. Priya smiled. “Don’t build an oracle. Build a research assistant that shows its work, says when it would be wrong, never trades by itself, and is tested honestly. That’s harder, and it’s a far better hackathon project.”

The brief

Build a tool that, given a stock ticker and an as-of date, produces a TradeIdea: a structured proposal (long, short or, often, no trade) with a thesis, evidence citations pointing at real retrieved news and computed signals, the key risks, an invalidation price at which the idea is clearly wrong, a time horizon and a confidence level.

The tool must only use information that existed on the as-of date. A person reviews every idea, and approved ideas go into a paper-trading journal. A replay over past dates measures how the ideas would have done, with an honest account of what such a test can and cannot prove.

Three ideas from earlier modules come together here. Agentic RAG (an agent that decides for itself what to retrieve, looks at the results, and searches again if needed, from the agents module) gathers evidence. Structured outputs with Pydantic (a Python library that checks data matches a declared shape) turn the answer into a typed object. And the guardrail principle from this module, “the model suggests, code decides”, governs everything that touches risk.

What you will build

Must have

• Daily price data for a small watchlist (five stocks), saved locally.
• Technical signals computed as of any past date: moving averages, RSI, volatility, average true range.
• A vector store of dated news, searchable only up to the as-of date.
• A TradeIdea Pydantic schema with thesis, evidence citations, risks, invalidation, horizon and confidence.
• An agent with three tools that researches and writes one idea per ticker and date.
• Deterministic risk checks in code: valid citations, sensible invalidation, position size computed by code.
• A review step where a person approves or rejects each idea, and a paper-trading journal.
• A replay that scores ideas on past dates, compared with a simple baseline.

Nice to have

• A Streamlit page for reviewing ideas.
• Charts of each idea’s entry, invalidation and target against later prices.
• Tracing every agent run with Langfuse.

Architecture

flowchart LR
  P["Price data"] --> S["Signals as of date"]
  N["News feed"] --> V["Vector store with dates"]
  S --> A["Research agent"]
  V --> A
  A --> T["TradeIdea JSON"]
  T --> R["Risk checks in code"]
  R --> H["Human review"]
  H --> J["Paper journal"]
  J --> E["Replay and evaluation"]

Price data. Daily open, high, low, close and volume for each ticker, downloaded once and saved as files so you are not hammering a data source during development.

Signals as of date. Functions that compute indicators using only rows up to the as-of date. This box is where many backtests quietly cheat, so it gets its own milestone.

News feed and vector store. Headlines and short articles, each with a ticker, a source, a URL and a publication timestamp. They are chunked, embedded (turned into lists of numbers that capture meaning) and stored in Chroma with the timestamp as metadata, so searches can be filtered to “published on or before the as-of date”.

Research agent. A Claude agent with three tools: get_signals, search_news and get_price_history. The as-of date is fixed inside the tools by your code, so the model cannot request the future, however it phrases a request.

TradeIdea JSON. The agent’s research notes are turned into a validated TradeIdea object.

Risk checks. Plain Python rules the model cannot argue with: every citation must exist, the invalidation level must make sense, and position size is calculated from fixed risk rules.

Human review and paper journal. A person reads each idea and its evidence, approves or rejects it, and approved ideas are logged with imaginary money.

Replay and evaluation. The system is run on many past dates; later prices decide whether each idea hit its target, its invalidation level, or neither.

Tech stack

Piece	Tool	Why
Language and packages	Python 3.12 with uv	Same setup as the rest of the course
Price data	yfinance (or NSE’s daily bhavcopy files)	Free and quick for learning; unofficial, so cache everything
Data wrangling	pandas	Rolling windows and date indexing make signals short
Vector store	Chroma	Local, simple metadata filters for dates and tickers
Model and agent	Anthropic SDK, claude-opus-5-5, tool runner	Tool use plus typed outputs with messages.parse
Schemas	Pydantic	The TradeIdea contract between the model and your code
Review UI (optional)	Streamlit	A review screen in about 40 lines
Tracing (optional)	Langfuse	See every tool call the agent made for an idea

Build it, milestone by milestone

1. Set up the project and its safety rails

Why. Safety rails are easiest to build first and hardest to add later. You are deciding, before any finance code exists, that this tool cannot trade.

uv init trade-research && cd trade-research
uv add anthropic chromadb pandas yfinance pydantic python-dotenv
uv add --dev pytest

Create config.py:

# config.py: the rules of the game, in one place
DISCLAIMER = "Educational project. Not investment advice. Paper trading only."
WATCHLIST = ["TCS.NS", "INFY.NS", "HDFCBANK.NS", "RELIANCE.NS", "ITC.NS"]
PAPER_CAPITAL_INR = 10_00_000        # imaginary ₹10 lakh
RISK_PER_IDEA = 0.005                 # risk at most 0.5% of capital per idea
MODEL = "claude-opus-5-5"

Then add a test that keeps execution code out of the project for good:

# tests/test_no_execution.py: this project must never be able to trade
from pathlib import Path

FORBIDDEN = ("kiteconnect", "place_order", "submit_order", "smartapi", "alpaca")

def test_no_broker_code():
    for path in Path(".").rglob("*.py"):
        if ".venv" in path.parts or path.name == "test_no_execution.py":
            continue
        text = path.read_text(encoding="utf-8").lower()
        for word in FORBIDDEN:
            assert word not in text, f"{path} mentions {word}"

Working looks like: uv run pytest passes, and DISCLAIMER is printed at the top of every script you write from now on.

Most likely bug: the test scanning thousands of files inside .venv and taking ages. The .venv check above prevents it.

Stuck? Hint

If rglob is slow, restrict it to your own folders, for example Path("src").rglob("*.py"), and keep all project code under src/. The point is not that a test makes trading impossible, but that adding it would require deliberately deleting a safety test, which is visible in review.

2. Ingest daily market data

Why. Everything else, signals, entries, invalidation levels and evaluation, depends on clean daily prices. Download once, save, and work from files.

# ingest_prices.py: download and cache daily prices
from pathlib import Path
import yfinance as yf
from config import WATCHLIST, DISCLAIMER

print(DISCLAIMER)
Path("data/prices").mkdir(parents=True, exist_ok=True)

for ticker in WATCHLIST:
    df = yf.Ticker(ticker).history(start="2023-01-01", end="2026-10-31", auto_adjust=True)
    df.index = df.index.tz_localize(None).normalize()   # plain dates, no time zones
    df = df[["Open", "High", "Low", "Close", "Volume"]]
    df.to_csv(f"data/prices/{ticker}.csv")
    print(f"{ticker}: {len(df)} rows, {df.index.min().date()} to {df.index.max().date()}")

auto_adjust=True adjusts past prices for splits and dividends, so a 1-for-2 split does not look like a 50% crash. .NS marks a National Stock Exchange of India listing.

Working looks like: about 950 rows per ticker (roughly 250 trading days a year), saved as CSV files.

Most likely bug: gaps and duplicates. Exchange holidays are normal gaps; but check for duplicate dates, zero volumes and sudden jumps of more than 20% in a day, and look at them by hand before trusting the data. yfinance is an unofficial library that reads Yahoo Finance’s website, so it occasionally breaks or returns odd data. Library APIs change quickly. If a name has moved, check the library’s documentation.

Stuck? Hint

If downloads fail, use a CSV you download by hand (NSE publishes daily “bhavcopy” files of every stock’s prices) and load it with pandas.read_csv(..., parse_dates=["Date"], index_col="Date"). The rest of the project only needs a DataFrame with a date index and the five columns.

3. Compute signals as of a date

Why. The single most common way a trading backtest lies is look-ahead bias: using information at a decision time that was not available yet. The defence starts here: every signal function takes an as_of date and cuts the data before computing anything.

# signals.py: indicators that can only see the past
import pandas as pd

def load_prices(ticker: str) -> pd.DataFrame:
    return pd.read_csv(f"data/prices/{ticker}.csv", index_col=0, parse_dates=True)

def signals_as_of(prices: pd.DataFrame, as_of: str) -> dict:
    df = prices.loc[:as_of]                       # nothing after the as-of date exists
    close, high, low = df["Close"], df["High"], df["Low"]

    delta = close.diff()                          # 14-day RSI: strength of recent gains vs losses
    gain = delta.clip(lower=0).rolling(14).mean()
    loss = (-delta.clip(upper=0)).rolling(14).mean()
    rsi = 100 - 100 / (1 + gain / loss)

    prev_close = close.shift(1)                   # average true range: typical daily move
    true_range = pd.concat([high - low, (high - prev_close).abs(),
                            (low - prev_close).abs()], axis=1).max(axis=1)

    return {
        "date": str(df.index[-1].date()),
        "close": round(close.iloc[-1], 2),
        "sma_20": round(close.rolling(20).mean().iloc[-1], 2),
        "sma_50": round(close.rolling(50).mean().iloc[-1], 2),
        "rsi_14": round(rsi.iloc[-1], 1),
        "atr_14": round(true_range.rolling(14).mean().iloc[-1], 2),
        "volatility_20d_pct": round(close.pct_change().rolling(20).std().iloc[-1] * 100, 2),
        "pct_from_52w_high": round((close.iloc[-1] / close.tail(252).max() - 1) * 100, 1),
    }

A moving average (SMA) is the average closing price over the last N days; price above a rising 50-day average is a common sign of an uptrend. RSI runs from 0 to 100; above 70 is often read as “stretched upwards”, below 30 as “stretched downwards”. ATR is the typical size of a day’s move, which you will use to judge whether an invalidation level is sensible.

Then write the test that proves the cut works:

# tests/test_signals.py
import pandas as pd
from signals import load_prices, signals_as_of

def test_signals_ignore_the_future():
    prices = load_prices("TCS.NS")
    real = signals_as_of(prices, "2025-06-02")
    tampered = prices.copy()
    tampered.loc["2025-06-03":, "Close"] *= 10     # wreck every price after the date
    assert signals_as_of(tampered, "2025-06-02") == real

Working looks like: a dictionary of eight numbers for any date, and the test passing.

Most likely bug: computing signals on the full DataFrame and then picking the row for the as-of date. Rolling windows are backward-looking, so that is usually fine, but any centred window, any shift(-1), or any normalisation over the whole series (like dividing by the all-time maximum) quietly leaks the future. The tampering test catches all of these.

Stuck? Hint

If the test fails, print signals_as_of for both versions and find which key differs. If the as-of date is a holiday, prices.loc[:as_of] correctly ends on the previous trading day; that is the behaviour you want.

4. Ingest and index dated financial news

Why. Signals say what the price did; news suggests why, and what might come next. But news is also the easiest way to leak the future, so every chunk carries its publication time, and every search filters on it.

Use any source you are allowed to store: company announcements from the exchange website, RSS feeds from financial news sites, or the hackathon’s provided dataset. Save each item as a JSON line with id, ticker, published_at (with time zone), source, url and text.

# news_store.py: store news with its publication time as a number
import json
from datetime import datetime
import chromadb

db = chromadb.PersistentClient(path="./db")
news = db.get_or_create_collection("news")

def to_ts(iso: str) -> int:
    return int(datetime.fromisoformat(iso).timestamp())   # iso must include a time zone

def index(path: str) -> None:
    items = [json.loads(line) for line in open(path, encoding="utf-8")]
    news.upsert(                                           # upsert: safe to re-run
        ids=[item["id"] for item in items],
        documents=[item["text"] for item in items],
        metadatas=[{"ticker": item["ticker"], "source": item["source"], "url": item["url"],
                    "published_at": item["published_at"],
                    "published_ts": to_ts(item["published_at"])} for item in items],
    )
    print(news.count(), "news chunks indexed")

def search_news_as_of(query: str, ticker: str, as_of_ts: int, k: int = 4) -> list[dict]:
    result = news.query(
        query_texts=[query], n_results=k,
        where={"$and": [{"ticker": ticker}, {"published_ts": {"$lte": as_of_ts}}]},
    )
    return [{"id": i, "text": d, **m} for i, d, m in
            zip(result["ids"][0], result["documents"][0], result["metadatas"][0])]

if __name__ == "__main__":
    index("data/news.jsonl")          # run once: uv run news_store.py

Chroma filters metadata with operators like $lte (less than or equal), so a search for “results guidance” as of 16 October cannot return an article from 17 October. Store the timestamp as a number, because comparisons on date strings are fragile.

Working looks like: a search with an as-of date before an article’s publication does not return it; the same search a day later does.

Most likely bug: time zones. A results announcement at 4:30 p.m. India time on 16 October is 11:00 UTC. If your as-of moment is “16 October” at midnight, that article is in the future. Decide when your decisions happen (in this project, 6 p.m. India time on the as-of date, after the market has closed) and convert every timestamp consistently.

Stuck? Hint

Write a test like the one in milestone 3: index two fake articles dated 15 and 17 October, search as of 16 October, and assert only the first comes back. If both come back, print the stored published_ts values; usually one was parsed without a time zone.

5. Define the TradeIdea schema

Why. The schema is the contract between the model and the rest of your code. It forces the model to separate thesis from evidence, to name what would prove it wrong, and to say “no trade” when nothing is compelling.

# schema.py: what a trade idea must contain
from datetime import date
from typing import Literal
from pydantic import BaseModel, Field

class Evidence(BaseModel):
    source_id: str = Field(description="A news chunk id or 'signal:<name>' exactly as returned by a tool")
    claim: str = Field(description="What this piece of evidence shows, in one sentence")

class TradeIdea(BaseModel):
    ticker: str
    as_of: date
    direction: Literal["long", "short", "no_trade"]
    thesis: str = Field(description="Two to four sentences: why this might work")
    evidence: list[Evidence] = Field(description="Every factual claim, each with its source")
    entry_price: float = Field(description="Closing price on the as-of date")
    invalidation_price: float = Field(description="Price at which the thesis is clearly wrong")
    target_price: float | None = Field(default=None, description="Where the idea would be taken off")
    time_horizon_days: int = Field(description="Trading days the idea should play out within")
    key_risks: list[str] = Field(description="What could go wrong, specific to this stock")
    confidence: Literal["low", "medium", "high"]

Three design choices are worth explaining. Confidence is a word, not a number. A model writing “0.73” suggests a precision nobody has; three levels are honest and easy to check against outcomes later. no_trade is a first-class answer. A research assistant that always finds a trade is a slot machine. Evidence is a list of IDs plus claims, so code can verify every citation in milestone 7.

Working looks like: TradeIdea.model_json_schema() prints a schema, and constructing a TradeIdea by hand with a missing field raises a validation error.

Most likely bug: putting rules such as “invalidation below entry for longs” only in field descriptions. Descriptions guide the model; they do not enforce anything. Enforcement happens in milestone 7.

Stuck? Hint

Build one valid TradeIdea by hand in a test, with made-up numbers, before involving the model. It forces you to decide what a good idea looks like, and becomes a fixture for the risk-check tests.

6. Build the agentic RAG research loop

Why. A fixed pipeline (“retrieve four chunks, then answer”) cannot follow a lead. An agent can notice that RSI is stretched, search for the news that explains it, find a guidance cut, and then check the price history around that date. That loop, deciding what to retrieve next based on what it has seen so far, is what makes it agentic RAG.

The key design: tools are created for one ticker and one as-of date, so the date lives in your code, not in the model’s arguments. The tools also record every ID they hand out, so citations can be checked later.

# agent.py: research one ticker as of one date
import json
from datetime import date, datetime, time
from zoneinfo import ZoneInfo
import anthropic
from anthropic import beta_tool
from config import MODEL
from schema import TradeIdea
from signals import load_prices, signals_as_of
from news_store import search_news_as_of

client = anthropic.Anthropic()
IST = ZoneInfo("Asia/Kolkata")

def make_tools(ticker: str, as_of: str, seen_ids: set[str]):
    prices = load_prices(ticker)
    # Research happens at 6 p.m. IST on the as-of date, after that day's close.
    decision_ts = int(datetime.combine(date.fromisoformat(as_of), time(18, 0), IST).timestamp())

    @beta_tool
    def get_signals() -> str:
        """Technical signals for the stock, computed from prices up to the research date's close."""
        sig = signals_as_of(prices, as_of)
        seen_ids.update(f"signal:{name}" for name in sig)
        return json.dumps(sig)

    @beta_tool
    def search_news(query: str) -> str:
        """Search news about the stock published before the research time.

        Args:
            query: What to look for, e.g. "quarterly results guidance".
        """
        hits = search_news_as_of(query, ticker, decision_ts)
        seen_ids.update(h["id"] for h in hits)
        return json.dumps([{k: h[k] for k in ("id", "published_at", "source", "text")} for h in hits])

    @beta_tool
    def get_price_history(days: int) -> str:
        """Daily closing prices for the last N trading days up to the research date.

        Args:
            days: Number of trading days, between 5 and 120.
        """
        days = max(5, min(days, 120))
        closes = prices.loc[:as_of, "Close"].tail(days).round(2)
        return closes.to_json(date_format="iso")

    return [get_signals, search_news, get_price_history]

Now the two-phase run: research freely with tools, then turn the notes into a typed idea.

RESEARCH_SYSTEM = """You are a cautious equity research assistant for an educational project.
Use the tools to gather evidence. Every factual claim must come from a tool result.
Prefer "no_trade" when evidence is thin, mixed or stale. You never execute trades."""

def research(ticker: str, as_of: str, max_steps: int = 8) -> tuple[TradeIdea, set[str]]:
    seen_ids: set[str] = set()
    runner = client.beta.messages.tool_runner(
        model=MODEL, max_tokens=4000, system=RESEARCH_SYSTEM,
        tools=make_tools(ticker, as_of, seen_ids),
        messages=[{"role": "user", "content":
                   f"Research {ticker} for a possible trade idea as of {as_of}. "
                   "Write research notes citing tool ids (news ids, signal:<name>)."}],
    )
    final = None
    for step, message in enumerate(runner):    # each item is one model turn
        final = message
        if step + 1 >= max_steps:              # a hard cap on loop length and cost;
            break                              # notes may then be partial, so expect no_trade
    notes = "\n".join(b.text for b in final.content if b.type == "text")

    idea = client.messages.parse(
        model=MODEL, max_tokens=2000, system=RESEARCH_SYSTEM,
        messages=[{"role": "user", "content":
                   f"Ticker {ticker}, as of {as_of}. Allowed evidence ids: {sorted(seen_ids)}.\n"
                   f"Turn these research notes into a TradeIdea:\n\n{notes}"}],
        output_format=TradeIdea,
    ).parsed_output
    return idea, seen_ids

Working looks like this run, on an illustrative fictional stock (the numbers are made up for teaching, not real market data). Research for DECCAN as of 16 October:

1. get_signals() returns close ₹812.40, 50-day average ₹768.10, RSI 71.2, ATR ₹18.60, 2.1% below its 52-week high.
2. The agent, noticing a stretched RSI, calls search_news("recent results and guidance"). It gets two chunks: n-4471 (14 Oct, “Q2 revenue up 14%, margin guidance raised”) and n-4466 (9 Oct, “promoter pledge reduced to zero”).
3. It calls get_price_history(30) to see whether the rise came before or after the results. Most of the rise followed 14 October.
4. It stops and writes notes. The parsed idea: long, entry ₹812.40, invalidation ₹774 (below the post-results gap), target ₹870, horizon 20 trading days, confidence medium, evidence n-4471, signal:sma_50, signal:rsi_14, and a risk noting “RSI above 70: entry after a sharp move, pullback risk”.

A run like this takes 4 model calls, about 25,000 tokens and roughly ₹10 to ₹15.

Most likely bug: the agent citing IDs it never received, such as news-123, or paraphrasing an ID. That is why seen_ids exists: milestone 7 rejects any idea whose citations are not in it.

Stuck? Hint

If the tool runner raises errors about tool definitions, check that each tool function has a docstring and type hints: the decorator builds the tool’s description and input schema from them. If the agent loops without stopping, lower max_steps and add “Stop after at most five tool calls” to the system prompt. Library APIs change quickly; if tool_runner or beta_tool has moved, check the Anthropic docs.

7. Add risk checks, human review and a paper journal

Why. The model is persuasive, and in markets persuasion is dangerous. Every rule that limits risk lives in code. The model never decides how much to buy; your code computes it from fixed rules.

# risk.py: rules the model cannot argue with
from config import PAPER_CAPITAL_INR, RISK_PER_IDEA
from schema import TradeIdea

def check(idea: TradeIdea, seen_ids: set[str], atr: float) -> list[str]:
    problems = []
    cited = {e.source_id for e in idea.evidence}
    if unknown := cited - seen_ids:
        problems.append(f"cites sources never retrieved: {sorted(unknown)}")
    if idea.direction == "no_trade":
        return problems
    if not any(s.startswith("signal:") for s in cited) or not any(not s.startswith("signal:") for s in cited):
        problems.append("needs at least one signal and one news source")
    long = idea.direction == "long"
    risk_per_share = (idea.entry_price - idea.invalidation_price) * (1 if long else -1)
    if risk_per_share <= 0:
        problems.append("invalidation is on the wrong side of entry")
    elif not (1 * atr <= risk_per_share <= 4 * atr):
        problems.append(f"invalidation {risk_per_share:.1f} away; expected 1 to 4 ATR ({atr:.1f})")
    if idea.target_price is not None and risk_per_share > 0:
        reward = (idea.target_price - idea.entry_price) * (1 if long else -1)
        if reward / risk_per_share < 1.5:
            problems.append(f"reward to risk {reward / risk_per_share:.2f}, below 1.5")
    if idea.confidence == "high" and len(idea.evidence) < 3:
        problems.append("high confidence needs at least three pieces of evidence")
    return problems

def paper_size(idea: TradeIdea) -> int:
    """Shares to buy with imaginary money, so a hit to invalidation loses 0.5% of capital."""
    risk_per_share = abs(idea.entry_price - idea.invalidation_price)
    return int(PAPER_CAPITAL_INR * RISK_PER_IDEA // risk_per_share)

For the DECCAN idea: risk per share is 812.40 − 774 = ₹38.40, which is about 2.1 ATR (fine); reward is ₹57.60, so reward to risk is exactly 1.5, the minimum, so it passes; the paper size is ₹5,000 ÷ ₹38.40 = 130 shares, about ₹1.06 lakh of imaginary money.

Then the review step. A simple command-line version prints the disclaimer, the idea, every piece of evidence with its full source text, and the risk-check results, and asks for approve, reject or skip with a reason. Approved ideas are appended to journal.csv with the date, ticker, direction, entry, invalidation, target, size, the reviewer’s reason, and a status of open. Ideas that failed risk checks are shown too, marked as rejected by code, because seeing why they failed is educational.

Working looks like: an idea with a fake citation is rejected automatically; a valid idea shows its evidence text side by side with the claims; the journal gains one row per approval.

Most likely bug: reviewers rubber-stamping. If you approve everything, the human step is theatre. Track your approval rate; if it is above about 80%, either the agent is excellent or you are not reading.

Stuck? Hint

Write unit tests for check using hand-made TradeIdea objects: one with an unknown citation, one long idea with invalidation above entry, one with a tiny target. These tests need no model calls, so they belong in the CI checks job from the previous lesson.

8. Backtest honestly and evaluate the agent

Why. Backtesting means running a strategy on past data to see how it would have done. It is the only way to learn anything before risking money, and it is also the easiest place in all of finance to fool yourself.

The replay: pick evaluation dates, run research for each ticker and date, apply the risk checks, and score each surviving idea using prices after the as-of date. An idea “wins” if price reaches the target before the invalidation level within the horizon, “loses” if it hits invalidation first, and “expires” otherwise. Measure results in R, multiples of the risk taken: a win at 1.5 reward to risk is +1.5R, a loss is −1R.

# replay.py: score one idea with prices the agent never saw
def score(idea, prices) -> float:
    future = prices[prices.index > str(idea.as_of)].head(idea.time_horizon_days)
    risk = abs(idea.entry_price - idea.invalidation_price)
    for _, day in future.iterrows():
        if idea.direction == "long":
            if day["Low"] <= idea.invalidation_price:
                return -1.0
            if idea.target_price and day["High"] >= idea.target_price:
                return (idea.target_price - idea.entry_price) / risk
        else:
            if day["High"] >= idea.invalidation_price:
                return -1.0
            if idea.target_price and day["Low"] <= idea.target_price:
                return (idea.entry_price - idea.target_price) / risk
    exit_price = future["Close"].iloc[-1]
    move = (exit_price - idea.entry_price) * (1 if idea.direction == "long" else -1)
    return move / risk

Compare against a baseline, for example “go long whenever price is above its 50-day average, with the same invalidation and target rules”, run on the same dates. If the agent cannot beat a two-line rule, the AI is not adding anything yet.

Now the caveats. These are not footnotes; they are the most important part of the milestone.

• The model may already know the future. An LLM trained on text up to some date has read news and price commentary about the past. Replaying 2024 with a model trained on 2025 data is like testing a student with an exam they have already seen. Only dates after the model’s training cutoff give a clean test; treat anything earlier as an optimistic upper bound.
• Look-ahead bias in your own data. Revised data, news timestamps in the wrong time zone, or scoring an idea with prices from the same day it was made all leak the future. Your tests from milestones 3 and 4 guard the obvious cases.
• Survivorship bias. Today’s watchlist contains companies that did well enough to still be large. Testing only on today’s winners flatters any strategy.
• Costs. Brokerage, taxes such as India’s securities transaction tax, and slippage (getting a worse price than the one you planned) can turn a small edge into a loss. Subtract a realistic cost per trade.
• Tiny samples. Forty ideas is not enough to tell skill from luck. Report the number of ideas next to every result.
• Overfitting the prompt. If you tweak the prompt until the backtest looks good, you have fitted the past, not learned anything about the future. Keep some dates aside and look at them only once, at the end.

Then evaluate the research, not only the returns. For a sample of 20 ideas, check by hand: does each claim match its cited source? Are the risks specific? Was “no trade” chosen when evidence was mixed? These are evals in the sense of the evaluations module, and they matter more for a learning project than P&L.

Working looks like: a short report with the number of ideas, win, loss and expiry counts, average R after costs, the same for the baseline, and a written paragraph on which caveats apply to your result.

Most likely bug: scoring with the as-of day’s own high and low. The prices.index > as_of filter starts on the next trading day, because the agent decides after the as-of day’s close and could not have traded earlier.

Stuck? Hint

Replays cost money: 5 tickers × 40 dates at about ₹12 per run is around ₹2,400. Start with 2 tickers and 10 dates to debug, cache every TradeIdea to disk keyed by ticker and date, and never re-run the agent for a date you have already scored.

Done means done

• DISCLAIMER `DISCLAIMER` appears on every script output and screen, and no broker or execution code exists (the test passes).
• Prices for five tickers are cached locally and checked for duplicates and suspicious jumps.
• Signals are computed as of a date, and the tampering test proves future prices cannot change them.
• News search never returns an item published after the decision time, with a test to prove it.
• The agent produces valid TradeIdea objects, including no_trade The agent produces valid `TradeIdea` objects, including `no_trade`, with citations only from retrieved sources.
• Risk checks reject bad citations, wrong-side or unreasonable invalidation levels, and poor reward to risk, with unit tests.
• Position sizes are computed by code from fixed risk rules, never by the model.
• Every idea passes through a human review step, and approved ideas land in a paper-trading journal.
• A replay report compares the agent with a simple baseline, after costs, and states its caveats in plain words.

Stretch goals

• Walk-forward evaluation: score each month using only ideas generated before that month, and chart results over time.
• Calibration check: do “high confidence” ideas actually win more often than “low”? If not, the confidence field is decoration.
• Contradiction search: add a tool that deliberately searches for evidence against the thesis, and require the idea to address it.
• Tracing: send every run to Langfuse so reviewers can see exactly which searches the agent ran.
• Daily paper run: a scheduled job (GitHub Actions on a timer works) that researches the watchlist every morning before the market opens, with results only in the journal.

Common pitfalls

• Trusting a fluent thesis. Well-written reasoning is not evidence. Read the cited sources, every time.
• Letting the model pick dates or sizes. The as-of date and position sizing belong to your code.
• Testing on dates the model has already “read about”. Results before the training cutoff are contaminated.
• Forgetting costs and slippage. Small edges vanish after realistic costs.
• Treating a good backtest as permission. A backtest is a hypothesis, not a promise. This project stops at paper trading.

Show it off

README. Open with the disclaimer in bold. Then one paragraph on what the agent does, the architecture diagram, a sample TradeIdea in JSON, and a section called “How I avoided look-ahead bias” that links to your two tests. Close with the replay report, including the caveats.

Demo. A 60-second screen recording: pick a ticker and date, watch the tool calls stream past, see the structured idea with its evidence, watch the risk checks pass or fail, and approve it into the paper journal.

What to write on LinkedIn. Lead with the engineering, not the returns. For example: “For FinHack Pune I built a research agent that proposes structured, cited trade ideas for human review. The hard part wasn’t the AI. It was making sure the agent could never see the future: date-locked tools, time-zone-safe news search, and tests that tamper with future prices to prove they’re ignored. Paper trading only, educational only.” Recruiters for AI roles care far more about that sentence than about a backtest chart.

Arjun and Kabir did not win FinHack. They came second, and the judges’ feedback singled out one thing: theirs was the only team that could explain why their backtest might be wrong. Priya read the final README on Monday. “This,” she said, tapping the look-ahead section, “is what an AI engineer sounds like.”