Streaming with Server-Sent Events

Three weeks before Diwali, Meera called Arjun into her office and turned her laptop around. On the screen was Crumb’s support chat widget, the one he had built over the last few modules. She typed “Can I change the delivery date on my Diwali hamper?” and pressed Enter.

Nothing happened. A small grey spinner turned. Four seconds. Eight. At eleven seconds a full, polite, perfectly correct paragraph appeared all at once.

“It’s a good answer,” Meera said. “But I closed the tab at six seconds.” She opened a second window: Priya’s analytics dashboard. Over the last week, 38 out of every 100 chats had been abandoned before the bot replied. The average reply took 11.2 seconds. “That’s a lot of customers walking out of the shop while the cake is still in the oven. Why does it make me wait and then dump everything on me?”

Arjun opened his code. He was already thinking about how to make the model faster. Priya, reading over his shoulder, had a different idea. “The model starts writing in under a second,” she said. “You’re hiding that from the customer.”

Arjun’s first attempt

Arjun’s endpoint looked like this. It is the same pattern you wrote in Module 05:

@app.post("/chat")
def chat(req: ChatRequest):
    response = client.messages.create(
        model="claude-opus-5-5",
        max_tokens=2000,
        system="You are Crumb's friendly support assistant.",
        messages=[{"role": "user", "content": req.message}],
    )
    return {"answer": next(b.text for b in response.content if b.type == "text")}

messages.create waits until the model has written its entire reply, then returns it. Only then does FastAPI send anything to the browser.

He remembered from Module 01 that the SDK (the provider’s helper library) could stream, so he tried that first:

@app.post("/chat")
def chat(req: ChatRequest):
    answer = ""
    with client.messages.stream(model="claude-opus-5-5", max_tokens=2000,
                                messages=[{"role": "user", "content": req.message}]) as stream:
        for text in stream.text_stream:
            print(text, end="", flush=True)  # look, it streams!
            answer += text
    return {"answer": answer}

In his terminal, words poured out one after another. He grinned and refreshed the widget. Eleven seconds of spinner, then the whole paragraph.

“It streams,” Priya said, “from Anthropic to your server. Then your server sits on it like a hen on an egg until the last word arrives, and hands the browser one big JSON blob. You fixed the first half of the pipe.”

The bakery counter at seven in the morning

Priya drew the Koregaon Park store’s counter on the whiteboard. “Croissants come out in trays, forty trays a morning. Do we wait for all forty before opening the doors?”

“No. You put each tray out as it’s ready.”

“When does the first customer start being happy?”

“When they get the first croissant.” Arjun saw where this was going. “Not when the fortieth tray comes out.”

“So there are two numbers,” Priya said, and wrote them down:

• Time to first token (TTFT): how long until the first piece of the answer appears.
• Total time: how long until the last piece appears.

An LLM (a large language model, the text-predicting system behind the bot) writes one token at a time. A token is a small chunk of text, roughly three-quarters of an English word, as you saw in Module 01. A 280-word reply is about 370 tokens. The model might produce the first one 0.9 seconds after receiving the question and the last one 11 seconds later.

“Streaming doesn’t make the model faster,” Priya said. “Total time stays at eleven seconds. What changes is the latency the customer feels.” Latency is the delay between asking and getting something back. “If the first words appear after one second, the customer starts reading. Most people read slower than the model writes, so they never notice the other ten seconds.”

Show the answer

Still about 11 seconds. Streaming does not speed up generation at all; the model writes at the same pace. It changes when the customer starts seeing progress. That is why it fixes abandonment (people leave when nothing seems to be happening) but would not help a batch job that only cares about the finished text.

Three ways to push words to a browser

“Okay,” Arjun said. “So the server needs to keep sending small pieces down to the browser. How?”

Priya listed three ways a customer could find out whether their cake is ready:

1. Call the bakery every minute and ask. This is polling: the browser asks the server “anything new?” again and again. Simple, but wasteful, and the answer always arrives up to one interval late.
2. Keep a phone line open in both directions. This is a WebSocket: a two-way connection where either side can talk at any moment. Powerful, and right for live games, voice and collaborative editing. It also needs special handling in load balancers and its own reconnection logic.
3. Tune into the bakery’s radio announcements. You ask once, then you listen while they talk. This is Server-Sent Events.

Server-Sent Events (SSE) is a web standard where the browser makes one ordinary HTTP request and the server answers with a response that stays open, writing small text “events” until it is done. It is one-way, server to browser, which is exactly the shape of a chat reply: the question goes up once, the answer streams down. Because it is plain HTTP, it works with the logins, cookies, proxies and load balancers you already have.

“Where the radio analogy breaks,” Priya added, “is that every listener gets their own private broadcast. One request, one stream, one customer.”

Here is the surprising part: you have been using SSE since Module 01 without knowing it. When the SDK streams from Anthropic or OpenAI, their servers send exactly this format. Arjun ran a raw request with curl to see it, and the first lines looked like this:

event: message_start
data: {"type":"message_start","message":{"id":"msg_01...","model":"claude-opus-5-5",...}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Yes"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":", you can change"}}

stream.text_stream in the SDK reads these events and hands you only the text pieces. Arjun’s job was to do the same thing one more hop: from his server to the customer’s browser.

Following one question from keyboard to screen

Before writing code, Priya made him trace a single question through the whole system, with real timings from a test run. The question was Meera’s: “Can I change the delivery date on my Diwali hamper?”

Time	Where	What happens
0.00 s	Browser	Sends POST /chat with {"message": "Can I change..."}
0.04 s	FastAPI	Receives the request, opens a stream to Anthropic
0.04 s	FastAPI	Sends response headers to the browser: status 200, Content-Type: text/event-stream. The connection stays open
0.91 s	Anthropic	First content_block_delta arrives at FastAPI with text "Yes"
0.91 s	FastAPI	Writes data: {"text": "Yes"} plus a blank line, 22 bytes, to the browser
0.93 s	Browser	Parses the event, appends “Yes” to the chat bubble. The customer sees words.
0.91 to 11.1 s	All three	About 90 more small events, each a few tokens of text
11.15 s	Anthropic	message_stop: the model is done
11.15 s	FastAPI	Writes event: done and closes the response
11.16 s	Browser	Hides the typing indicator

The test reply used 46 input tokens (system prompt plus question) and 371 output tokens. At Claude Opus 5.5’s prices at the time of writing ($4permillioninputtokensand$20 per million output tokens), that is 0.000184 + 0.00742 = about $0.0076, roughly ₹0.67 (using about ₹88 to the dollar). The non-streaming version cost exactly the same. Streaming changes the experience, not the bill.

Notice the third row. The server sends the status code at the start, long before it knows whether the model will finish successfully. That detail will matter in a moment.

Opening the hood: what an SSE stream looks like

An SSE response is plain text. Here is the stream Arjun’s server will send for the question above:

data: {"text": "Yes"}

data: {"text": ", you can change"}

data: {"text": " the date up to 48 hours before delivery."}

event: done
data: {}

The rules fit on an index card:

• Each event is one or more lines of the form field: value, ended by one blank line. The blank line is what tells the browser “this event is complete”.
• data: carries the payload. Most apps put a small JSON object here.
• event: names the event (done, error). Without it, the browser treats the event as a plain “message”.
• id: labels an event so a reconnecting browser can say where it got up to, and retry: tells the browser how many milliseconds to wait before reconnecting.
• A line starting with a colon, such as : ping, is a comment. Browsers ignore it. Servers use it as a heartbeat to keep quiet connections alive.
• The response’s Content-Type header must be text/event-stream.

Under the hood, the server writes these bytes onto an HTTP response that it does not close. Each write is flushed across the network immediately rather than collected. That is all streaming is: a response you keep writing to.

Show the answer

The browser sees data: Yes! followed by a line Here's how: that does not start with a known field name, so that second line is ignored and the text is lost. Worse, if the model ever writes two newlines in a row (a paragraph break), the browser treats it as the end of the event. Wrapping each piece in JSON turns the newline into the two characters \n inside a string, so the event stays on one line and arrives intact.

Building the server, one layer at a time

Layer 1: a stream with no AI at all

Start with the simplest thing that can stream: a generator that yields words with a pause between them. A generator is a function that hands out values one at a time with yield instead of returning once. FastAPI’s StreamingResponse sends each yielded value to the browser the moment it is produced.

uv add fastapi anthropic python-dotenv

# main.py
import asyncio

from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()


@app.get("/demo")
async def demo():
    async def event_stream():
        for word in ["Fresh", " croissants", " coming", " out", " now"]:
            yield f"data: {word}\n\n"  # one event, ended by a blank line
            await asyncio.sleep(0.5)   # pretend the model is thinking

    return StreamingResponse(event_stream(), media_type="text/event-stream")

Run it with uv run fastapi dev main.py, then in another terminal:

curl -N http://localhost:8000/demo

The -N flag turns off curl’s own buffering. You should see the lines appear half a second apart:

data: Fresh

data:  croissants

data:  coming
...

If they appear one by one, your streaming pipe works end to end. Everything after this is filling the pipe with real content.

Layer 2: safe events

Add a helper that formats every event with JSON, so newlines in model output cannot break it:

import json


def sse(data: dict, event: str | None = None) -> str:
    """Format one Server-Sent Event: optional name, JSON data, blank line."""
    lines = [f"event: {event}"] if event else []
    lines.append(f"data: {json.dumps(data)}")
    return "\n".join(lines) + "\n\n"

sse({"text": "Yes"}) returns 'data: {"text": "Yes"}\n\n'. sse({}, event="done") returns 'event: done\ndata: {}\n\n'.

Layer 3: real model output

Now replace the fake words with the model. Use AsyncAnthropic, the asynchronous client, so that one slow stream does not block every other customer’s request while it waits:

import anthropic
from dotenv import load_dotenv
from pydantic import BaseModel

load_dotenv()  # reads ANTHROPIC_API_KEY from .env
client = anthropic.AsyncAnthropic()


class ChatRequest(BaseModel):
    message: str


@app.post("/chat")
async def chat(req: ChatRequest):
    async def event_stream():
        try:
            async with client.messages.stream(
                model="claude-opus-5-5",
                max_tokens=2000,
                system="You are Crumb's friendly support assistant.",
                messages=[{"role": "user", "content": req.message}],
            ) as stream:
                async for text in stream.text_stream:
                    yield sse({"text": text})  # forward each piece immediately
            yield sse({}, event="done")
        except anthropic.APIError:
            yield sse({"message": "Sorry, something went wrong. Please try again."}, event="error")

    return StreamingResponse(
        event_stream(),
        media_type="text/event-stream",
        headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
    )

Line by line, the new parts:

• async with client.messages.stream(...) opens the stream to Anthropic. The async for loop receives each text piece as Anthropic sends it.
• yield sse({"text": text}) forwards that piece to the browser before the next one arrives. This is the line Arjun’s first attempt was missing.
• The try/except handles failures during the stream. Remember the status code was sent at 0.04 seconds as 200. If the model fails at second 6, you cannot change it to 500 any more. So you send an error event and let the browser show a message. This is called reporting the error in-band: inside the stream itself.
• Cache-Control: no-cache stops caches from holding on to the response. X-Accel-Buffering: no tells the Nginx web server, which sits in front of many apps, not to collect the whole response before forwarding it.

Test it:

curl -N -X POST http://localhost:8000/chat \
  -H "Content-Type: application/json" \
  -d '{"message": "Can I change the delivery date on my Diwali hamper?"}'

data: {"text": "Yes"}

data: {"text": ", you can change the delivery date"}

data: {"text": " up to 48 hours before"}
...
event: done
data: {}

Layer 4: the browser

Browsers ship a built-in EventSource object that reads SSE, parses events and even reconnects automatically. It has one big limitation: it can only make GET requests, with no request body. A chat message belongs in a POST body. So most AI apps read the stream with fetch and do the parsing themselves:

async function ask(message, onText) {
  const res = await fetch("/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ message }),
  });
  const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();
  let buffer = "";
  while (true) {
    const { value, done } = await reader.read();
    if (done) break;
    buffer += value;
    const events = buffer.split("\n\n");
    buffer = events.pop(); // the last piece may be half an event: keep it for next time
    for (const evt of events) {
      const dataLine = evt.split("\n").find((l) => l.startsWith("data: "));
      if (!dataLine) continue; // a comment or heartbeat
      const data = JSON.parse(dataLine.slice(6));
      if (evt.startsWith("event: error")) return onText("\n[" + data.message + "]");
      if (evt.startsWith("event: done")) return;
      onText(data.text);
    }
  }
}

The buffer is the important trick. The network does not promise to deliver one event per chunk. A single read() might return half an event, or three and a half. So you collect text, split on the blank line, handle every complete event and keep the unfinished tail for the next read. Forgetting this works perfectly on your laptop and fails randomly on a slow mobile network.

If you ever stream something that only needs a GET, such as /orders/1042/progress, EventSource is shorter: new EventSource(url).onmessage = (e) => show(e.data).

Where it breaks: the staging surprise

Arjun deployed to the staging server on Thursday afternoon. The widget showed a spinner for eleven seconds, then the whole answer. Exactly the old behaviour.

It worked on his laptop. It worked with curl against his laptop. On staging, it arrived in one lump. Priya asked one question: “What’s between the browser and FastAPI on staging?”

Nginx. Nginx buffers responses by default: it collects what the app sends and forwards it in large pieces, which is efficient for web pages and fatal for streams. The X-Accel-Buffering: no header was there, but a teammate’s config also enabled a compression middleware, which collected the stream to compress it as one block. Removing compression on the /chat route fixed it.

Here are the other places streams break in production:

• Idle timeouts. Load balancers often close connections that send nothing for 30 to 60 seconds. If the model takes a long time before its first token (a long prompt, or a model that thinks before answering), the connection can die before anything arrives. Send a : ping comment every 15 seconds while waiting.
• Errors after the first byte. You saw this above. Every client must handle an error event, or a failure mid-answer looks like the bot stopped talking forever.
• Retries. A request that fails before the first token can be retried safely. One that fails halfway has already shown the customer half an answer. The next lesson deals with this.
• The six-connection limit. Over older HTTP/1.1, browsers allow only about six open connections per website. A customer with seven Crumb tabs each holding an open stream will see the seventh hang. HTTP/2, which most modern hosts use, removes this limit.

Try it yourself

1. Build Layer 1 and confirm with curl -N that words arrive half a second apart.
2. Build Layer 3 and ask “Write four lines about Diwali sweets.”
3. Now break it on purpose: change the endpoint to collect every piece into a list and yield it all at the end. Time both versions with a stopwatch, measuring time to first word and time to last word.

Show the answer

The streaming version prints its first data: line within about a second and keeps printing until it finishes, a few seconds later. The collected version prints nothing for the whole generation time, then everything at once. The time to last word is nearly identical in both. Only the time to first word changed, and that is the number your customers feel.

Common mistakes

• Streaming into your server but not out of it. Like Arjun’s first attempt: the SDK streams, then you collect and return JSON. Yield each piece onward.
• Forgetting the blank line. Each event must end with \n\n. With one \n, the browser waits for an event that never finishes.
• Raw text in data:. One newline in model output splits or truncates your event. JSON-encode every payload.
• A buffering layer in the middle. Nginx, compression middleware and some hosting platforms collect the response. Test streaming on the real deployment, not only on localhost.
• Using EventSource for POST. It cannot send a body. Use fetch with a reader and a buffer.
• No error event. The status code is already 200; failures must travel inside the stream.

Recap

• Streaming does not make the model faster; it shows the answer as it is produced, cutting the time to first token from seconds to under one.
• Server-Sent Events is one long HTTP response of small text events, each ending in a blank line, served as text/event-stream.
• In FastAPI, return a StreamingResponse wrapping an async generator that forwards each model piece as a JSON-encoded event.
• Errors after the first byte must be sent in-band, as an error event.
• In the browser, read POST streams with fetch, a reader and a buffer; EventSource is for GET only.
• Test streaming through every layer in production: proxies, compression and idle timeouts all break it.

On Friday, Meera tried the widget again. The first words appeared before she had finished lifting her chai. “Explain to me what you changed,” she said, “without the word ‘token’.”

Arjun thought for a moment. “Before, the kitchen baked the whole order, boxed it and only then opened the hatch. Now we pass each piece through the hatch the moment it’s ready. The order takes the same time. But you start eating at one second instead of eleven, so you don’t walk out.” Meera nodded. The abandonment rate the next week was 9 percent.

Then Priya wrote a new number on the whiteboard: 429. “Diwali traffic is about to be five times normal,” she said. “The provider will start saying no. Next, we teach the app to wait politely and try again.”

Go deeper

• FastAPI documentation, including custom and streaming responses: https://fastapi.tiangolo.com
• Anthropic streaming and API reference: https://docs.anthropic.com
• OpenAI API documentation (streaming section): https://platform.openai.com/docs