Routing And Streaming

router(), split(), retry(), and fallback() can stream. cascade() is generate-only.

const streamResult = await stream(chatPrompt, {
  model: router({
    id: "chat-router",
    classify: () => "default",
    routes: { default: fallback([primary, backup]) },
  }),
  input,
});

Fallback can retry a stream only before the first token reaches the caller. timeout.firstToken is designed for this:

fallback([primary, backup], {
  timeout: { firstToken: 3000, attempt: 10000 },
});

After token 1, Crux does not restart the stream with another model. The stream errors, and the completion error carries the routing receipt with midStreamFailure: true on the fallback step. The receipt's firstTokenAt remains the elapsed milliseconds to that first emitted token for successful completions and post-token failures alike.

try {
  for await (const delta of streamResult.textStream) {
    send(delta);
  }
  await streamResult.completion;
} catch (error) {
  console.log(error.routing);
}