Streaming does not make the model faster. It stops the reader staring at nothing, which is most of the experience.
Highlights
- Streaming does not reduce how long a response takes, and measured here the total was 7.9 seconds either way.
- What changes is when the first word appears, which fell from 7.9 seconds to 0.1, a wait 78 times shorter for identical work.
- Server sent events are a plain text format over an ordinary HTTP response, so they need no new protocol and no library.
- One blank line separates events, and forgetting the second newline is the most common reason a stream silently never arrives.
- Anything that buffers between your server and the browser turns a stream back into a blocking request, and nothing reports an error when it happens.
- A reasoning model streams its thinking first, so code watching only the answer field shows an empty screen while tokens are flowing.
- Errors happen after the response has already started, so a status code cannot carry them and you need an error event instead.
A model that takes eight seconds to answer feels broken. The same model, streaming, feels fast.
Nothing about it got faster. Same machine, same model, same prompt. Both took 7.9 seconds. One showed a spinner the whole time. The other started writing after 0.1 seconds.
That is the whole value of streaming. Worth being precise about, because it tells you when to build it and when not to bother.
What you need before you start
Everything here runs on one machine. A Linux box, which a KodeKloud playground gives you, or your own laptop.
Nothing in this post needs an API key. The model runs locally, so you can break things freely without spending anything.
Install Ollama
sudo apt-get update && sudo apt-get install -y zstd python3 python3-venv
curl -fsSL https://ollama.com/install.sh | shThe zstd line is not optional on a fresh image. The installer extracts a zstd archive and stops with an error that never names the cause. python3-venv is separate from python3 on Debian and Ubuntu, and without it the next section fails.
Check it is serving
Ollama is running means the installer registered a systemd service and started it, which is what happens on a normal virtual machine. A connection refused means there is no systemd, common in containers, so start it yourself:
ollama serve &Running that when it is already up gives you address already in use, which reads like a failure and is really the port being held by the copy that is already working.
Pull a model
ollama pull llama3.2:1bThat is 1.3GB and runs on almost anything. Any chat model works, and a small one is better here because a slow model makes the difference streaming makes more obvious.
Set up Python
mkdir streaming && cd streaming
python3 -m venv .venv
source .venv/bin/activateNo packages to install. Everything in this post uses the standard library.
You will create four files in that directory as you go.
Step 1: Measure the wait you are removing
Create compare.py. It runs the same prompt twice, once blocking and once streaming, warming the model first so no load cost is involved.
import json
import time
import urllib.request
OLLAMA = "http://localhost:11434/api/generate"
MODEL = "llama3.2:1b"
PROMPT = "Explain in about eighty words why a Linux service might fail to start."
def request(stream, num_predict=100):
body = json.dumps({
"model": MODEL,
"prompt": PROMPT,
"stream": stream,
"options": {"num_predict": num_predict, "temperature": 0},
}).encode()
return urllib.request.Request(
OLLAMA, data=body, headers={"Content-Type": "application/json"}
)
def blocking():
started = time.time()
urllib.request.urlopen(request(False)).read()
return time.time() - started
def streaming():
started = time.time()
first = None
with urllib.request.urlopen(request(True)) as response:
for line in response:
if not line.strip():
continue
chunk = json.loads(line)
if (chunk.get("response") or chunk.get("thinking")) and first is None:
first = time.time() - started
if chunk.get("done"):
break
return first, time.time() - started
urllib.request.urlopen(request(False, num_predict=5)).read() # warm the model
time.sleep(1)
total = blocking()
time.sleep(1)
first, stream_total = streaming()
print(f"{MODEL}, both runs warm\n")
print(f" blocking request user sees nothing for {total:.1f}s, then everything")
print(f" streaming request first token at {first:.1f}s, done at {stream_total:.1f}s")
print(f"\n Same work, same duration. The wait before anything appears fell")
print(f" from {total:.1f}s to {first:.1f}s, which is {total / first:.0f} times shorter.")Run it:
python3 compare.pyThe total is identical. The wait before anything appears fell from 7.9 seconds to 0.1, which is 78 times shorter.
So streaming is not a performance fix. It turns one long silence into steady evidence that something is happening. That distinction matters when you decide where to spend effort.
The first row surprises people. Streaming a fast response adds moving parts for no gain, and a batch job nobody watches gains nothing at all.
What server sent events actually are
SSE sounds like a protocol. It is really a text format, sent over an ordinary HTTP response that stays open.
You set one header, then write plain text blocks until you are done. Here is what goes over the wire.
That is the whole format. Three rules cover almost everything.
Each line is field: value. You will use event for a name and data for the payload. There are also id and retry, which you rarely need.
A blank line ends the event. So each block finishes with two newlines, not one.
Anything else is ignored. That is why a line starting with a colon works as a keepalive comment.
Watch out
Forgetting the second newline is the most common reason a stream produces nothing, and it fails silently. Your event needs to end with \n\n rather than \n. With only one, the browser holds the event open, waiting for a boundary that never comes, so EventSource fires no handler and no error. Everything looks connected and nothing arrives. If a stream is silent, check that before anything else.
Why not WebSockets
SSE and WebSockets solve different problems, and streaming a response is squarely SSE's.
Tokens flow one way, so the extra power of WebSockets goes unused while its costs do not. Automatic reconnection alone is worth it, since browsers retry a dropped EventSource with no code from you.
Step 2: Stream tokens from the model
Create stream_tokens.py. Ollama streams when you ask it to, and each line of the response is a JSON object.
import json
import urllib.request
OLLAMA = "http://localhost:11434/api/generate"
MODEL = "llama3.2:1b"
def token_stream(prompt):
body = json.dumps({
"model": MODEL,
"prompt": prompt,
"stream": True,
"options": {"num_predict": 120, "temperature": 0.3},
}).encode()
request = urllib.request.Request(
OLLAMA, data=body, headers={"Content-Type": "application/json"}
)
with urllib.request.urlopen(request) as response:
for line in response:
if not line.strip():
continue
chunk = json.loads(line)
if chunk.get("response"):
yield chunk["response"]
if chunk.get("done"):
returnA generator is the right shape here. Each token is yielded as it arrives. Nothing piles up in memory, and the caller decides what to do with each piece.
Check it works before building anything on top:
python3 -c "from stream_tokens import token_stream; [print(t, end='', flush=True) for t in token_stream('Say three short words.')]"Words should appear one at a time rather than all together. If they arrive in a clump, the model is producing faster than you can see, which is fine and will still be visible in the browser.
Watch out
A reasoning model streams its thinking before its answer, so code watching only the answer field sees an empty screen while tokens are flowing. Measured here, a reasoning model produced its first token at 0.2 seconds, and every one of those tokens was thinking rather than answer. With a hundred token budget it used the whole allowance thinking and never reached the answer at all. The chunks carry a separate thinking field, so if you want the honest picture, watch both and decide whether to show the thinking or just a signal that it is happening.
Step 3: Serve the tokens as events
Create server.py. It imports the generator you just wrote and wraps it in an HTTP response the browser understands. No framework, just the standard library.
import json
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import urlparse, parse_qs
from stream_tokens import token_stream
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
url = urlparse(self.path)
if url.path != "/stream":
self.send_error(404)
return
prompt = parse_qs(url.query).get("q", ["Say hello."])[0]
self.send_response(200)
self.send_header("Content-Type", "text/event-stream")
self.send_header("Cache-Control", "no-cache")
self.send_header("Connection", "keep-alive")
self.send_header("X-Accel-Buffering", "no")
self.end_headers()
try:
for piece in token_stream(prompt):
payload = json.dumps({"text": piece})
self.wfile.write(f"event: token\ndata: {payload}\n\n".encode())
self.wfile.flush()
self.wfile.write(b"event: done\ndata: {}\n\n")
self.wfile.flush()
self.close_connection = True
except BrokenPipeError:
pass
except Exception as exc:
payload = json.dumps({"message": str(exc)[:200]})
self.wfile.write(f"event: error\ndata: {payload}\n\n".encode())
self.wfile.flush()
self.close_connection = True
def log_message(self, *args):
pass
if __name__ == "__main__":
ThreadingHTTPServer(("", 8000), Handler).serve_forever()Five details in there are each load bearing.
Content-Type: text/event-stream is what makes the browser treat this as a stream rather than a document.
self.wfile.flush() after every write pushes the bytes out now. Without it Python buffers, and tokens arrive in clumps or all at the end.
X-Accel-Buffering: no tells nginx not to buffer this response. Everything else ignores it. One line, and it avoids a problem that is hard to diagnose.
BrokenPipeError is caught and ignored. A user closing the tab mid response is normal, not exceptional. Without this, your logs fill with tracebacks from people navigating away.
Errors become an event rather than a status code. By the time something fails you have already sent 200 OK, so the only way to tell the client is in the stream itself.
self.close_connection = True ends the response. Without it the server keeps the socket open after the final event, waiting for another request on the same connection. A browser does not care, because EventSource closes from its side on the done event. Anything else hangs, and your server accumulates a thread per abandoned connection.
Start it in one terminal and leave it running:
python3 server.pyStep 4: Check the tokens are not being buffered
Create timing.py. This is the check to keep, because it is how you tell a working stream from a buffered one.
import time
import urllib.request
URL = "http://localhost:8000/stream?q=name+three+reasons+a+linux+service+fails+to+start"
started = time.time()
arrivals = []
with urllib.request.urlopen(URL) as response:
for raw in response:
if raw.startswith(b"data:"):
arrivals.append(time.time() - started)
if raw.startswith(b"event: done"):
break
gaps = [later - earlier for earlier, later in zip(arrivals, arrivals[1:])]
instant = sum(1 for gap in gaps if gap < 0.001)
print(f" {len(arrivals)} events | first at {arrivals[0]:.2f}s | last at {arrivals[-1]:.2f}s")
print(f" median gap {sorted(gaps)[len(gaps) // 2] * 1000:.0f}ms "
f"| gaps under 1ms: {instant} of {len(gaps)}")
if instant > len(gaps) * 0.8:
print("\n Everything arrived at once. Something between here and the")
print(" server is buffering, so streaming is off even though it looks on.")
else:
print("\n Tokens are arriving one at a time, so nothing is buffering.")With the server still running, in a second terminal:
python3 timing.pyA median gap of 96 milliseconds means they really are being pushed as generated. If those gaps were near zero and the total unchanged, everything would be arriving at the end. That is the failure the next section covers.
Step 5: Read the stream in a browser
The client side is smaller than the server side. Add this page to server.py, or open the complete version at the end of this post, which already includes it.
<!DOCTYPE html>
<html>
<head><meta charset="utf-8"><title>Streaming</title></head>
<body>
<input id="q" size="60" value="Name three reasons a Linux service fails to start.">
<button id="go">Ask</button>
<pre id="out"></pre>
<script>
const out = document.getElementById("out");
let source = null;
document.getElementById("go").addEventListener("click", () => {
if (source) source.close();
out.textContent = "";
const q = encodeURIComponent(document.getElementById("q").value);
source = new EventSource(`/stream?q=${q}`);
source.addEventListener("token", (e) => {
out.textContent += JSON.parse(e.data).text;
});
source.addEventListener("done", () => {
source.close();
});
source.addEventListener("error", (e) => {
out.textContent += "\n\n[stream failed]";
source.close();
});
});
</script>
</body>
</html>EventSource does the work. It opens the connection, splits events on blank lines, and dispatches them by name.
Two behaviours are worth knowing before they surprise you.
It reconnects on its own. If the connection drops, the browser retries after a few seconds. Usually what you want, occasionally not. Calling close() stops it, which is why the done handler exists.
Its error event is not your error event. The browser fires error for network problems. It also fires when the server closes the connection normally. So calling close() on done stops an error handler firing on every success.
Practise on a machine you can throw away
Streaming is easier to understand when you can watch it break. The KodeKloud playgrounds give you a Linux box, so you can run Ollama and this server together, put a proxy in front, and see what buffering does.
The failure that looks like success
Here is the one that costs people an afternoon.
Anything between your server and the browser can buffer the response. When it does, your server streams perfectly, the client gets everything at the end, and no error is raised anywhere. Streaming is off and nothing says so.
The usual culprits, and what each needs:
Diagnosing it is easier than it sounds, because the failure has a distinctive shape. Time the arrival of each event. If the gaps are near zero and the total time is unchanged, it all arrived at once, and something between you and the browser is holding it.
That test takes ten lines and tells you immediately whether the problem is your code or your infrastructure.
In the wild
Streaming usually works in development and breaks on the first deploy, because development has nothing in front of the server. Locally you connect straight to your process. In production there is a reverse proxy, probably a load balancer, possibly a CDN, and each is entitled to buffer. So the first real test of a streaming endpoint is behind the same stack it will run behind, not on your laptop. Testing it locally proves your code works, which was rarely the thing in doubt.
Handling failures mid stream
Streaming changes what an error means, and this is the part most tutorials skip.
An ordinary request either succeeds or fails, and the status code says which. A stream sends 200 OK before it knows whether the work will finish. So anything failing afterwards cannot use a status code.
That leaves three cases worth handling.
The model fails partway. Send an error event so the client can show what happened rather than silently stopping mid sentence.
The client disappears. They closed the tab. Catch BrokenPipeError, stop generating, and move on. On a paid API this also saves money, since you stop the upstream request rather than paying for tokens nobody will read.
The connection idles. Some proxies close a connection with no traffic for thirty seconds. If your model might think for longer than that, send a comment line as a keepalive.
self.wfile.write(b": keepalive\n\n")
self.wfile.flush()A line starting with a colon is a comment in SSE. The browser ignores it. The proxy sees traffic, so the connection stays open.
The whole thing, in one file
Everything above, ready to run.
import json
import urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import urlparse, parse_qs
OLLAMA = "http://localhost:11434/api/generate"
MODEL = "llama3.2:1b"
PAGE = """<!DOCTYPE html>
<html><head><meta charset="utf-8"><title>Streaming</title></head>
<body>
<input id="q" size="60" value="Name three reasons a Linux service fails to start.">
<button id="go">Ask</button>
<pre id="out"></pre>
<script>
const out = document.getElementById("out");
let source = null;
document.getElementById("go").addEventListener("click", () => {
if (source) source.close();
out.textContent = "";
const q = encodeURIComponent(document.getElementById("q").value);
source = new EventSource(`/stream?q=${q}`);
source.addEventListener("token", e => { out.textContent += JSON.parse(e.data).text; });
source.addEventListener("done", () => source.close());
source.addEventListener("error", () => { out.textContent += "\\n\\n[stream failed]"; source.close(); });
});
</script>
</body></html>"""
def token_stream(prompt):
body = json.dumps({
"model": MODEL,
"prompt": prompt,
"stream": True,
"options": {"num_predict": 120, "temperature": 0.3},
}).encode()
request = urllib.request.Request(
OLLAMA, data=body, headers={"Content-Type": "application/json"}
)
with urllib.request.urlopen(request) as response:
for line in response:
if not line.strip():
continue
chunk = json.loads(line)
if chunk.get("response"):
yield chunk["response"]
if chunk.get("done"):
return
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
url = urlparse(self.path)
if url.path == "/":
body = PAGE.encode()
self.send_response(200)
self.send_header("Content-Type", "text/html; charset=utf-8")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
return
if url.path != "/stream":
self.send_error(404)
return
prompt = parse_qs(url.query).get("q", ["Say hello."])[0]
self.send_response(200)
self.send_header("Content-Type", "text/event-stream")
self.send_header("Cache-Control", "no-cache")
self.send_header("Connection", "keep-alive")
self.send_header("X-Accel-Buffering", "no")
self.end_headers()
try:
for piece in token_stream(prompt):
payload = json.dumps({"text": piece})
self.wfile.write(f"event: token\ndata: {payload}\n\n".encode())
self.wfile.flush()
self.wfile.write(b"event: done\ndata: {}\n\n")
self.wfile.flush()
self.close_connection = True
except BrokenPipeError:
pass
except Exception as exc:
payload = json.dumps({"message": str(exc)[:200]})
self.wfile.write(f"event: error\ndata: {payload}\n\n".encode())
self.wfile.flush()
self.close_connection = True
def log_message(self, *args):
pass
if __name__ == "__main__":
print("Open http://localhost:8000")
ThreadingHTTPServer(("", 8000), Handler).serve_forever()Run it, open the page, and you have a streaming chat in about ninety lines with no dependencies beyond Python and a local model.
Try this yourself
An hour, and you will recognise every one of these failures on sight.
Run the server and time the events arriving, so you know what working looks like on your machine.
Then break it on purpose. Remove the flush() calls and watch tokens clump. Remove one of the two newlines and watch the browser receive nothing at all, with no error anywhere.
Now put something in front of it. Even a ten line Python proxy that reads the whole upstream response before forwarding shows what a buffering layer does. That is the failure you are most likely to meet for real.
Finally, close the browser tab halfway through a response and check your server logs. Without the BrokenPipeError handler you get a traceback, which is how you learn that ordinary user behaviour was being logged as an error.
What you should be able to answer now
Does streaming make a response faster? No. Measured here the total was 7.9 seconds either way, and the wait before anything appeared fell from 7.9 seconds to 0.1.
Why do events end with two newlines? Because a blank line is what ends an event. With one newline the browser waits for a boundary that never arrives, and nothing fires, including the error handler.
Why can a stream fail with no error anywhere? Because something in the middle buffered it. Your server streams, the client gets everything at the end, and no layer reports a problem.
How do you signal a failure that happens mid stream? With an error event, since the 200 OK went out before you knew anything was wrong.
The measurement at the top is the reason to do any of this. Same model, same duration, and a wait that went from feeling broken to feeling instant. Everything after it is plumbing, and the plumbing is ninety lines.
Ready to Build on Top of This?
Streaming is one piece of making an AI feature feel finished. The AI Agents Fundamentals course covers tokens, embeddings, retrieval, orchestration and MCP through an end to end project. The AI Learning Path sequences that alongside vector databases and agents. Start by timing your own first token.
FAQs
Q1: Does streaming make an LLM respond faster?
No, and it helps to be precise. Measured on the same machine, same model, same prompt, both warm, a blocking request and a streaming one each took 7.9 seconds. The work is identical, because the model generates at the same rate either way. What changes is when the first word reaches the user. That fell from 7.9 seconds to 0.1, a wait 78 times shorter for identical work. So streaming is a perception change, not a performance one. That tells you where it is worth building. Under a second, it adds moving parts for no gain. Between three and ten seconds, it is the difference between an interface that feels broken and one that feels instant. For a batch job nobody watches, it buys nothing.
Q2: What are server sent events and how do they differ from WebSockets?
Server sent events are a text format, sent over an ordinary HTTP response that stays open. You set Content-Type: text/event-stream and write blocks of field: value lines, each ending with a blank line. No new protocol, and no library on either side, since browsers have EventSource built in. WebSockets solve a different problem. They go both ways, need a protocol upgrade, and make you write your own reconnection logic. A model response flows one way, so that extra power goes unused while its costs do not. SSE also travels as plain HTTP, so proxies and firewalls mostly leave it alone. Browsers reconnect a dropped stream on their own. For token streaming, SSE is simpler and fits better almost every time.
Q3: Why does my SSE stream arrive all at once instead of token by token?
Something between your code and the browser is buffering. It is hard to spot because nothing reports an error. Your server streams correctly, the client gets everything at the end, and every layer thinks it succeeded. There are four usual suspects. Python buffers unless you call flush() after each write. nginx buffers proxied responses unless you set proxy_buffering off or send the X-Accel-Buffering: no header. gzip buffers in order to compress, so exclude text/event-stream. And some CDNs buffer by default. To find it, time the arrival of each event. If the gaps are near zero and the total time is unchanged, it all arrived at once. Working correctly here, the median gap between tokens was 96 milliseconds.
Q4: How do I report an error that happens after streaming has started?
With an event, because a status code is no longer available. Streaming means sending 200 OK before you know whether the work will finish, so a failure two seconds later cannot change it. Define an error event, send it with a message, and handle it on the client. Two related cases are worth handling too. When a user closes the tab mid response, your writes raise BrokenPipeError. That is ordinary behaviour rather than a fault, so catch it and stop generating. On a paid API it also stops you paying for tokens nobody will read. And if a proxy might close an idle connection before your model produces anything, send a comment line. That is any line starting with a colon. The browser ignores it, and the proxy sees traffic.
Q5: Why does my stream produce nothing at all, with no error?
Check your line endings first. This is the most common cause, and it fails in total silence. Each SSE event must end with two newlines rather than one, because a blank line marks the end of an event. With a single newline, the browser holds the event open, waiting for a boundary that never comes. No handler fires and no error is raised. Everything looks connected and nothing arrives. The second thing to check is the event name. If your server sends event: token and your client listens for message, nothing matches. EventSource dispatches by name, and the default message handler only catches events with no name set. Both mistakes produce the same symptom, which is silence.
Q6: What do I need to build this?
Python and a model to stream from. Nothing else. The server here uses only the standard library. The browser side uses EventSource, which every modern browser has built in, so there is no framework on either side. For the model, a local one through Ollama needs no key and no quota, so experimenting is free. A hosted API works the same way, since most providers offer a streaming mode yielding chunks you forward identically. The pattern is the same whichever you use. Read tokens from a generator. Write each one as an SSE event. Flush after every write. Send a final event so the client knows to close. About ninety lines, including the HTML page.
Discussion