Langfuse, LangSmith, Braintrust and OpenLLMetry

By Thursday morning Arjun had a tracing tool running. He had found it in a developer newsletter, signed up for the free tier in four minutes, pasted an API key into .env, and wrapped CrumbBot’s chat handler with the vendor’s decorator. By lunchtime, 214 conversations had flowed into a slick dashboard in someone else’s cloud.

He showed Meera the screen with some pride. Every request was there as a tree. You could click into the retrieval step and see the four chunks.

Meera leaned in and read the first trace out loud. “Hi, it’s for my daughter Riya, she has a severe peanut allergy, please call me on 98220 41763 before delivery to Flat 4B, Shanti Kunj, Aundh.” She sat back. “Arjun. Where is this company? Who at that company can read this? How long do they keep it? And what does it cost when we’re not on the free tier?”

He did not know the answer to any of the four questions.

Priya, who had heard the last part, pulled up a chair. “Those are exactly the right questions. Let’s answer them properly, for four real tools, and then choose.”

Arjun’s first attempt, and what it leaked

Here is roughly what Arjun had shipped. The vendor name is changed, but the pattern is common.

# support.py (Thursday morning version)
from somevendor import trace_llm   # a hosted tracing SDK, free tier

@trace_llm(capture_input=True, capture_output=True)   # record everything
def handle_message(session_id: str, text: str) -> str:
    query = rewrite_query(text)
    chunks = retrieve(query, k=4)
    return generate_answer(text, chunks)

Nothing in that code is wrong as code. The problem is everything around it. In half a day, Crumb had copied 214 customer messages, including phone numbers, flat numbers and children’s medical conditions, to a company whose terms nobody had read. Remember Module 20: Meera had already been worried about sending allergy notes to three American AI companies. Arjun had added a fourth.

Cameras and the control room

Priya drew a shop on the whiteboard, with small boxes in the corners.

“Crumb’s Baner store has CCTV. What are the parts?”

“Cameras,” Arjun said. “And… the recorder in the back office. And the screen where the manager watches it.”

“Two jobs, then. The cameras capture. The recorder and screen store and show. You can buy them as one kit from one brand. Or you can buy cameras that speak a standard format and plug them into any recorder.”

She wrote two words under the drawing.

Instrumentation is the camera: code inside your app that creates spans. It can be a decorator you add by hand, or an auto-instrumentation library that patches the Anthropic or OpenAI SDK so every call becomes a span without you touching it.

A backend is the recorder and the control room: a service that receives spans, stores them, and gives you a user interface to search, filter and read trace trees. It may also add LLM-specific views: cost per trace, a chat-style view of prompts, user feedback scores.

“Where does the analogy break?” Arjun asked.

“CCTV footage is the same kind of thing whatever camera you buy. Trace data isn’t, yet. Each vendor’s own SDK records slightly different fields, with different names. That’s why a shared standard matters, and it’s what the next lesson is about. For now: some products sell you the whole kit, and one of the four we’re looking at is only cameras.”

Four tools, honestly

All four below can record CrumbBot’s tree: model calls, retrieval, tools, tokens, latency. The differences are in licence, hosting and what they are best at.

Langfuse is an open-source LLM engineering platform. Its core is MIT-licensed, which means you can read, run and modify the code for free; a few enterprise features sit under a separate commercial licence. You can self-host it with Docker on your own server, or use the hosted Langfuse Cloud. It covers tracing, sessions, cost tracking, prompt management (storing versioned prompts outside your code), datasets and evaluations with scores. Its Python SDK (since version 3) is built on OpenTelemetry, the open standard from Lesson 3, and the server also accepts spans from other OpenTelemetry tools.

LangSmith is made by the LangChain team. It is a hosted service, with self-hosting available on enterprise plans, and it is not open source. Its standout strength is deep integration with LangChain and LangGraph (Module 14): turn on an environment variable and every chain, node and tool call in a LangGraph agent appears as a tree. It also works without LangChain through a @traceable decorator. It covers tracing, datasets, evaluations and a prompt hub.

Braintrust is a hosted platform designed evaluation-first. Its centre of gravity is experiments: run your app over a dataset, score every output with scorers (functions or models that grade an answer), and compare runs side by side. It also logs and traces production traffic, and has a playground for trying prompts. The platform is not open source, though its autoevals scoring library is. Enterprise customers can keep the data in their own cloud account while Braintrust runs the interface.

OpenLLMetry, by Traceloop, is open source under the Apache 2.0 licence. It is instrumentation only: cameras, no control room. It is built on OpenTelemetry and auto-instruments OpenAI, Anthropic, vector databases like Chroma and Pinecone, and frameworks like LangChain. It then exports spans to any backend that speaks OpenTelemetry: Jaeger, Grafana Tempo, Honeycomb, Datadog, or Langfuse.

Others exist too, such as Arize Phoenix, Helicone and Weights & Biases Weave, and the market moves monthly.

	Langfuse	LangSmith	Braintrust	OpenLLMetry
What it is	Platform (instrumentation + backend)	Platform	Platform	Instrumentation library
Open source?	Yes, MIT core	No	No (scoring library is)	Yes, Apache 2.0
Self-host?	Yes, free, with Docker	Enterprise plans	Enterprise: data in your cloud	Not applicable; you pick the backend
Best at	All-round LLM tracing, sessions, costs, prompts	LangChain and LangGraph apps	Evaluations and experiments	Fitting AI into existing observability
Watch out for	Self-hosting means you run several databases	Data leaves your servers on standard plans	Evaluation-first; heavier than you need for tracing alone	No UI of its own; generic backends lack LLM views

The same brownie question, four ways

To compare fairly, Priya had Arjun replay Nandini’s brownie question from the last lesson through each tool, using test data only, with nothing personal in it. Here is what each showed for that one request: 3.21 seconds, two model calls, one retrieval.

In Langfuse, the trace opened as a tree: handle_message at the top, then rewrite_query and generate_answer marked as generations (Langfuse’s word for model-call spans), with retrieve between them. Clicking generate_answer showed the full prompt as a chat, 1,912 input tokens, 96 output tokens and a cost of $\mathrm{0.0096.}Thetraceheaderaddeditup:$0.0106, about ₹0.93 at roughly ₹88 to the dollar. A Sessions tab grouped all three of the test customer’s messages into one conversation.

In LangSmith, the same tree appeared as runs: a parent run with child runs typed as llm or chain. Token counts, latency and full inputs and outputs were all there. One click added the request to a dataset for later testing.

In Braintrust, the request appeared as one row in a project’s logs, expandable into the same spans. The interface kept nudging towards scoring: attach a scorer, add this to a dataset, run an experiment comparing top_k=4 with top_k=6.

With OpenLLMetry into Grafana Tempo, the request appeared as a waterfall: horizontal bars, one per span, lined up on a timeline, beautifully clear about where the 3.21 seconds went. Each model span carried attributes with names starting gen_ai. (Lesson 3 explains those). But there was no chat view of the prompt, no cost column and no session grouping out of the box.

“They all see the same request,” Arjun said. “But each one makes a different thing easy.”

“Right. Langfuse and LangSmith make reading traces easy. Braintrust makes testing changes easy. OpenLLMetry makes fitting in easy.”

What it costs, and where the data lives

Meera’s cost question has three kinds of answer, because tools charge in three ways. Exact prices change often, so check each tool’s pricing page; these are the shapes.

• Self-hosted open source. The software is free. You pay for the server, its storage and backups, and a person’s time to upgrade it. Data stays wherever your server is.
• Hosted, usage-based. A free tier, then a price per unit of volume: traces, spans or “events” per month, sometimes plus data retention. Costs grow with traffic.
• Hosted, per seat or enterprise. A monthly price per team member, or a negotiated contract that can include self-hosting or keeping data in your own cloud.

For CrumbBot, with clearly made-up round numbers: 1,800 requests a day at about 6 spans each is roughly 324,000 spans a month. On a hosted usage plan that might sit inside a free tier or cost a modest monthly fee, but it scales with every new customer. Self-hosting costs, say, ₹10,000 (about $114) a month for a server, plus an hour or two of Arjun’s time, whatever the volume. Run your real numbers against each pricing page.

The bigger question for Crumb was not money. It was Meera’s other question: who can read Riya’s allergy note?

Crumb’s decision

Priya laid out the reasoning on the whiteboard.

1. Privacy first. In Module 20, Crumb set up a back-office server to keep sensitive data in-house. Traces are some of the most sensitive data Crumb has. Self-hosting keeps them there.
2. LLM-specific views matter. Arjun needs to read prompts as conversations, see cost per trace, group sessions and attach customer feedback. A generic backend would make that hard.
3. Don’t lock the door behind you. Crumb’s backend team already runs OpenTelemetry and Grafana for the order service. A tool that speaks OpenTelemetry keeps that door open.

That pointed to Langfuse, self-hosted. LangSmith would have been a strong choice if CrumbBot were built on LangGraph; it isn’t. Braintrust would shine once the team does serious evaluation work, which starts in Module 23, but they needed tracing first. OpenLLMetry was attractive for fitting into Grafana, and they would come back to that idea in the next lesson.

Setting up Langfuse

Self-hosting uses Docker Compose, which you met in Module 20: one file that starts several containers together. Langfuse’s server needs a few supporting services (at the time of writing: Postgres, ClickHouse, Redis and S3-compatible storage), and the official compose file starts them all.

# On the back-office server (needs Docker; a few GB of free memory)
git clone https://github.com/langfuse/langfuse.git
cd langfuse
docker compose up -d
# Then open http://localhost:3000, create an account, a project and API keys

In the Langfuse interface, create a project called crumb-support and copy its keys into CrumbBot’s .env:

LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_BASE_URL=http://localhost:3000
ANTHROPIC_API_KEY=sk-ant-...

Now the smallest useful trace. Two functions, one inside the other, each decorated with @observe().

# uv add langfuse anthropic python-dotenv
import anthropic
from dotenv import load_dotenv
from langfuse import get_client, observe, propagate_attributes

load_dotenv()                  # reads the Langfuse and Anthropic keys from .env
langfuse = get_client()        # uses LANGFUSE_PUBLIC_KEY, _SECRET_KEY and _BASE_URL
client = anthropic.Anthropic()

@observe(as_type="generation")         # a model call: Langfuse shows tokens and cost
def ask_claude(question: str) -> str:
    response = client.messages.create(
        model="claude-opus-5-5",
        max_tokens=500,
        system="You are Crumb's friendly support assistant.",
        messages=[{"role": "user", "content": question}],
    )
    langfuse.update_current_generation(
        model="claude-opus-5-5",
        usage_details={"input": response.usage.input_tokens,
                       "output": response.usage.output_tokens},
    )
    return next(b.text for b in response.content if b.type == "text")

@observe()                             # a plain span; the outermost one becomes the trace
def handle_message(session_id: str, text: str) -> str:
    with propagate_attributes(session_id=session_id, tags=["support-chat"]):
        return ask_claude(text)

print(handle_message("sess-test-001", "Do you have eggless cakes?"))
langfuse.flush()   # send buffered spans before this short script exits

What each new piece does:

• @observe() turns a function into a span. It records the function’s arguments as the span’s input and its return value as the output. When one observed function calls another, the inner span becomes a child automatically, because Langfuse tracks the current span the same way the toy tracer in Lesson 1 did.
• as_type="generation" marks a model call, so Langfuse shows it with model, token and cost columns.
• update_current_generation attaches the model name and token usage from Claude’s response.usage. Langfuse computes cost from its table of model prices; for a model it doesn’t know yet, cost shows as zero until you add the price in its settings.
• propagate_attributes stamps trace-level fields on this span and every span created inside the with block: the session id groups this request with others in the same conversation.
• flush() sends anything still buffered. A web server running for days sends spans in the background; a script that exits in two seconds may quit before they leave.

These examples use the Langfuse Python SDK as it is at the time of writing. Older tutorials set the session with update_current_trace(...) and the server address with LANGFUSE_HOST; newer versions use propagate_attributes and LANGFUSE_BASE_URL. If you see the old names, that is why.

Run it with uv run, open the Langfuse interface, and you should see one trace named handle_message with a child generation ask_claude, about 40 input tokens and a cost of a fraction of a cent. Langfuse also ships a drop-in wrapper for the OpenAI SDK: change import openai to from langfuse.openai import openai and every call is traced.

The same shape in the other tools

For comparison, here is the same idea in OpenLLMetry and LangSmith. You will not need both; the point is how similar they look.

OpenLLMetry instruments the Anthropic SDK for you, so there is no decorator on the model call at all:

# uv add traceloop-sdk anthropic python-dotenv
import anthropic
from dotenv import load_dotenv
from traceloop.sdk import Traceloop

load_dotenv()
Traceloop.init(app_name="crumb-support")   # patches Anthropic, OpenAI, Chroma, ...

client = anthropic.Anthropic()
response = client.messages.create(          # this call becomes a span automatically
    model="claude-opus-5-5",
    max_tokens=500,
    messages=[{"role": "user", "content": "Do you have eggless cakes?"}],
)
print(next(b.text for b in response.content if b.type == "text"))

Where do the spans go? At the time of writing, by default to Traceloop’s own hosted platform if you set its API key, or to any OpenTelemetry endpoint you point it at with the TRACELOOP_BASE_URL environment variable (for example, a Langfuse server or Crumb’s Grafana stack). Check that setting before you run it against real customer data.

LangSmith uses a decorator called @traceable, and reads LANGSMITH_TRACING=true and LANGSMITH_API_KEY from the environment:

# uv add langsmith anthropic python-dotenv
import anthropic
from dotenv import load_dotenv
from langsmith import traceable

load_dotenv()
client = anthropic.Anthropic()

@traceable(run_type="llm")
def ask_claude(question: str) -> str:
    response = client.messages.create(
        model="claude-opus-5-5", max_tokens=500,
        messages=[{"role": "user", "content": question}])
    return next(b.text for b in response.content if b.type == "text")

@traceable
def handle_message(text: str) -> str:
    return ask_claude(text)

print(handle_message("Do you have eggless cakes?"))

Library APIs change quickly. If a name has moved, check the docs linked below.

Where the choice can go wrong

Lock-in through decorators. Every vendor SDK has its own decorators and field names. Sprinkle @trace_llm across 40 functions and switching tools later means touching all 40. One defence is a thin wrapper of your own (one traced() helper that calls the vendor), another is the shared standard in the next lesson.

Self-hosting is a real job. Langfuse self-hosted means Postgres, ClickHouse, Redis and object storage on a server you patch, back up and upgrade. If nobody owns that, the hosted version with masking may be safer than a neglected server.

Double tracing. Turn on OpenLLMetry’s auto-instrumentation and wrap the same calls with Langfuse’s decorator, and you can get every model call twice, with costs counted twice. Pick one source of spans per call.

Missing prices. A new model, like claude-opus-5-5 when it first came out, may not be in a tool’s price table yet. Your dashboard then shows ₹0 per trace, which is reassuring and wrong. Check one trace’s cost by hand against the provider’s pricing.

Try it yourself

Set up a local Langfuse and confirm the numbers yourself.

1. Start Langfuse with Docker Compose as above and create a project.
2. Run the @observe script three times with the same session_id and three different questions.
3. In the interface, open the Sessions view and find your session.
4. Open one generation. Compute its cost by hand at $4permillioninputtokensand$20 per million output tokens (the claude-opus-5-5 price at the time of writing) and compare with what Langfuse shows.

Expected result. One session containing three traces, each with a handle_message span and an ask_claude generation. A short question like “Do you have eggless cakes?” with the one-line system prompt is around 30 to 40 input tokens and 60 to 150 output tokens, so roughly $0.002to$0.003 per trace, about 20 to 25 paise. If Langfuse shows $0, add the model’s price in its settings.

Common mistakes

• Signing up before reading the data terms. Region, retention, data use and a data processing agreement come first, not after the first 214 traces.
• Treating OpenLLMetry as a complete tool. It records; it doesn’t store or show. You still need a backend, and you need to know which one it is sending to.
• Forgetting flush() in scripts and jobs. Spans vanish silently. No error tells you.
• Choosing on features alone. Most tools can draw a trace tree. Hosting, licence, privacy and how the bill grows decide more.
• Trusting cost columns blindly. Verify one trace’s cost by hand, especially for new models.

Recap

• Tracing has two jobs: instrumentation records spans in your code, a backend stores and shows them. Some tools do both; OpenLLMetry only records.
• Langfuse is open source (MIT core), self-hostable with Docker, and covers tracing, sessions, costs, prompts and evals. LangSmith is hosted, closed source and strongest with LangChain and LangGraph. Braintrust is hosted and evaluation-first.
• Before sending traces anywhere, know the data region, retention, data use and whether a data processing agreement exists.
• Pricing comes as self-hosted (pay for servers and time), usage-based, or per seat and enterprise. Check current pricing pages.
• Crumb chose self-hosted Langfuse for privacy and LLM views, while keeping the OpenTelemetry door open.

Meera listened to the decision and asked one thing. “So the allergy notes never leave our server now?”

“Not for tracing,” Arjun said. “Langfuse runs on the back-office machine. Only the four of us can log in, and traces are deleted after 30 days.”

Sneha from the backend team had been listening from the next desk. “Hang on. We already send every order-service span to Grafana through OpenTelemetry. When a customer says ‘the bot said my order was delivered but it wasn’t’, I want the bot’s spans and my order-service spans in one tree. Not in two tools.” Priya smiled. That was the next lesson.

Go deeper

• Langfuse documentation (self-hosting, Python SDK, sessions, costs): https://langfuse.com/docs
• LangSmith product page and docs links: https://www.langchain.com/langsmith
• Braintrust documentation: https://www.braintrust.dev/docs
• OpenLLMetry on GitHub (supported libraries and backends): https://github.com/traceloop/openllmetry