Building a Reliable Streaming AI Feature in Next.js
Your AI feature works great in demos. Then it meets the real internet.
You build a nice little feature: a button that sends text to an AI model, and the answer streams back word by word. It looks magical in a demo. But then you ship it, and here's what actually happens:
- The user clicks the button and waits... and waits...
- The model gives up halfway through and the UI just stares at you
- The API returns a 503 because someone else is also using it
- The model sends back garbage that crashes your parser
- The user clicks away and the old request keeps running in the background
Sound familiar? Streaming AI features are easy to build and surprisingly hard to ship well.
This article builds a small summarization endpoint in Next.js that handles all of that. By the end, you'll have:
- Input validation so users can't send a novel to the model
- Retries that don't make things worse
- A parser that treats the model's output like the untrusted data it is
- A UI that knows when the stream actually finished
The key insight is simple: keep the contract between your server and your browser smaller than the contract with the AI provider. Providers change. Your browser should only see three things: some text, a "done" signal, or an error.
The code is Next.js App Router + TypeScript. It works with any OpenAI-compatible API. The provider URL, model, and API key are environment variables, so you're not locked to one vendor.
What the browser needs to see
Before writing any code, decide what events the browser can observe:
| Event | What it means | What the browser does | | --- | --- | --- | | text | A chunk of validated text | Append it to the answer | | done | The stream finished normally | Show "Complete" | | error | Something went wrong | Show an error message |
This is not a copy of the provider's JSON. It's your own, simpler contract. If you change providers tomorrow, the browser doesn't need to change.
The setup
You need one dependency:
npm install zod
And three environment variables:
Step 1: Parse the stream without trusting it
Create lib/ai/stream.ts. This file turns the provider's messy SSE format into your clean three-event contract.
import { z } from 'zod'
const providerChunk = z.object({
choices: z.array(z.object({
delta: z.object({ content: z.string().optional() }).optional(),
})),
})
const retryableStatuses = new Set([408, 409, 425, 429, 500, 502, 503, 504])
const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms))
export const parseProviderFrame = (frame: string): string | 'done' | null => {
const data = frame
.split('\n')
.find((line) => line.startsWith('data:'))
?.slice(5)
.trim()
if (!data) return null
if (data === '[DONE]') return 'done'
const parsed = providerChunk.safeParse(JSON.parse(data))
if (!parsed.success) throw new Error('Provider returned an invalid stream event')
return parsed.data.choices[0]?.delta?.content || null
}
async function requestWithRetry(prompt: string): Promise<Response> {
const url = process.env.AI_API_URL
const apiKey = process.env.AI_API_KEY
const model = process.env.AI_MODEL
if (!url || !apiKey || !model) throw new Error('AI environment is incomplete')
for (let attempt = 0; attempt < 3; attempt += 1) {
const controller = new AbortController()
const timeout = setTimeout(() => controller.abort(), 20_000)
try {
const response = await fetch(url, {
method: 'POST',
signal: controller.signal,
headers: {
authorization: 'Bearer ' + apiKey,
'content-type': 'application/json',
},
body: JSON.stringify({
model,
stream: true,
temperature: 0,
messages: [
{
role: 'system',
content: 'Summarize the user text accurately in five short bullet points.',
},
{ role: 'user', content: prompt },
],
}),
})
if (response.ok || !retryableStatuses.has(response.status) || attempt === 2) {
return response
}
} catch (error) {
if (attempt === 2) throw error
} finally {
clearTimeout(timeout)
}
await sleep(250 * 2 ** attempt)
}
throw new Error('AI request failed')
}
export async function streamAnswer(prompt: string): Promise<ReadableStream<Uint8Array>> {
const upstream = await requestWithRetry(prompt)
if (!upstream.ok) throw new Error('AI provider rejected the request')
if (!upstream.body) throw new Error('AI provider returned no stream')
const reader = upstream.body.getReader()
const decoder = new TextDecoder()
const encoder = new TextEncoder()
return new ReadableStream({
async start(controller) {
let buffer = ''
let finished = false
let sawProviderDone = false
const emit = (value: Record<string, unknown>) => {
controller.enqueue(encoder.encode(JSON.stringify(value) + '\n'))
}
try {
while (!finished) {
const result = await reader.read()
buffer += decoder.decode(result.value || new Uint8Array(), { stream: !result.done })
let boundary = buffer.indexOf('\n\n')
while (boundary !== -1) {
const frame = buffer.slice(0, boundary)
buffer = buffer.slice(boundary + 2)
const value = parseProviderFrame(frame)
if (value === 'done') {
sawProviderDone = true
finished = true
break
}
if (value) emit({ type: 'text', value })
boundary = buffer.indexOf('\n\n')
}
if (result.done) {
if (buffer.trim()) {
const value = parseProviderFrame(buffer)
if (value === 'done') sawProviderDone = true
else if (value) emit({ type: 'text', value })
}
if (!sawProviderDone) throw new Error('AI stream ended before completion')
finished = true
}
}
if (!sawProviderDone) throw new Error('AI stream ended before completion')
emit({ type: 'done' })
controller.close()
} catch (error) {
emit({ type: 'error', message: 'The summary could not be completed.' })
controller.close()
} finally {
reader.releaseLock()
}
},
})
}
What's actually happening here
Provider sends: SSE frames (could be anything)
|
v
parseProviderFrame() | Validates with Zod
| Rejects garbage
v
Your stream: {type: 'text', value: '...'}
{type: 'done'}
{type: 'error', message: '...'}
The timeout is per attempt and fires before any data arrives. Once the stream has started, retrying would duplicate output — so we don't.
The buffer is necessary because network chunks don't respect line boundaries. One chunk might cut a JSON object in half. The parser carries the leftover between reads and tries again.
The error event sent to the browser is intentionally generic. Provider details belong in your server logs, not in the user's face.
AI_API_URL=https://api.openai.com/v1/chat/completions
AI_API_KEY=replace-me
AI_MODEL=replace-with-a-model-supported-by-your-provider
Keep the API key server-only. Don't use NEXT_PUBLIC_.
Step 2: Add the route
Create app/api/summarize/route.ts. Validate input at the route boundary, even if the client already does. Clients are not trusted.
import { z } from 'zod'
import { streamAnswer } from '@/lib/ai/stream'
const input = z.object({
prompt: z.string().trim().min(1).max(4_000),
})
export async function POST(request: Request) {
let body: unknown
try {
body = await request.json()
} catch {
return Response.json({ error: 'Request body must be valid JSON.' }, { status: 400 })
}
const parsed = input.safeParse(body)
if (!parsed.success) {
return Response.json({ error: 'prompt must be between 1 and 4,000 characters.' }, { status: 400 })
}
try {
const stream = await streamAnswer(parsed.data.prompt)
return new Response(stream, {
headers: {
'cache-control': 'no-store',
connection: 'keep-alive',
'content-type': 'application/x-ndjson; charset=utf-8',
'x-content-type-options': 'nosniff',
},
})
} catch (error) {
console.error('summarize request failed', error)
return Response.json({ error: 'The AI service is temporarily unavailable.' }, { status: 503 })
}
}
The route returns newline-delimited JSON, not raw SSE. That means your browser parser stays simple, and you can swap providers without touching the frontend.
Also notice: the response ends without a done event if something goes wrong. That's a feature — the browser knows the answer is incomplete.
Step 3: Build a client that knows its states
A text area and a string aren't enough. You need idle, streaming, complete, and error states. Here's the minimum:
'use client'
import { FormEvent, useState } from 'react'
export function Summarizer() {
const [prompt, setPrompt] = useState('')
const [answer, setAnswer] = useState('')
const [status, setStatus] = useState<'idle' | 'streaming' | 'complete' | 'error'>('idle')
async function submit(event: FormEvent) {
event.preventDefault()
setAnswer('')
setStatus('streaming')
try {
const response = await fetch('/api/summarize', {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify({ prompt }),
})
if (!response.ok || !response.body) throw new Error('request failed')
const reader = response.body.getReader()
const decoder = new TextDecoder()
let buffer = ''
let completed = false
while (true) {
const result = await reader.read()
buffer += decoder.decode(result.value || new Uint8Array(), { stream: !result.done })
const lines = buffer.split('\n')
buffer = lines.pop() || ''
for (const line of lines) {
if (!line) continue
const event = JSON.parse(line) as { type: string; value?: string }
if (event.type === 'text') setAnswer((current) => current + (event.value || ''))
if (event.type === 'error') throw new Error('stream failed')
if (event.type === 'done') completed = true
}
if (result.done) break
}
setStatus(completed ? 'complete' : 'error')
} catch {
setStatus('error')
}
}
return (
<form onSubmit={submit}>
<textarea value={prompt} onChange={(event) => setPrompt(event.target.value)} />
<button disabled={status === 'streaming' || !prompt.trim()}>Summarize</button>
<p aria-live='polite'>
{status === 'streaming' && 'Writing...'}
{status === 'complete' && 'Complete'}
{status === 'error' && 'Something went wrong. Try again.'}
</p>
<pre>{answer}</pre>
</form>
)
}
For production, add an AbortController to cancel the request if the user starts over or leaves. And if the model emits tiny deltas, buffer updates for 30-60ms to avoid re-rendering on every single token.
Step 4: Test the contract, not the model
The model is not a stable test dependency. Test the boundaries you control.
npm install -D vitest
import { describe, expect, it } from 'vitest'
import { parseProviderFrame } from './stream'
describe('parseProviderFrame', () => {
it('extracts a validated text delta', () => {
expect(parseProviderFrame('data: {"choices":[{"delta":{"content":"hello"}}]}')).toBe('hello')
})
it('recognizes the provider end marker', () => {
expect(parseProviderFrame('data: [DONE]')).toBe('done')
})
it('rejects malformed provider data', () => {
expect(() => parseProviderFrame('data: {"choices":null}')).toThrow()
})
})
You should also test: invalid JSON, empty prompt, oversized prompt, provider 503, and a stream that ends without done. The happy path is the least interesting test.
What can go wrong (and how to handle it)
Problem: Solution:
User sends 100,000 chars Validate max length at the route
Model returns 503 Retry 3x before the stream starts
Stream ends without [DONE] Throw an error, don't pretend it worked
User navigates away mid-stream AbortController cancels the request
Model sends malformed JSON Zod rejects it, you send an error event
Security and cost matter too
A few things that are easy to forget:
- Bound the input. Unbounded prompts cost money and time.
- Cap the output. Set a max token limit at the provider.
- Don't log user content. Summaries can contain personal data.
- Treat output as untrusted. Never insert it as HTML. Never use it in redirects.
- Rate limit. One user shouldn't be able to burn through your API budget.
Streaming makes the wait shorter, but it doesn't make the request cheaper.
What I would actually ship
- Validate and bound input at the server
- Retry only before output begins
- Parse provider events and validate their shape
- Translate them into your own simple stream format
- Track completion separately from accumulated text
- Test the unhappy paths
- Measure latency, retries, tokens, and errors
No queues. No agents. No vector databases. Those are fine later, but they don't fix a missing timeout or an unvalidated stream. Build the contract first.
Further reading: Next.js Route Handlers, MDN Streams API, MDN Server-sent events, Zod documentation, and the OWASP Top 10 for Large Language Model Applications.