Multimodal

Crux uses one media model throughout the SDK: usable media goes directly into an ordinary message. Storage is separate and optional.

Start with a normal message

import { prompt } from "@use-crux/core";
import { generate } from "@use-crux/openai";

const inspect = prompt({
  messages: () => [
    {
      role: "user",
      content: [
        { type: "text", text: "What is in this image?" },
        { type: "image", source: imageBytes, mediaType: "image/png" },
      ],
    },
  ],
});

const answer = await generate(inspect, { model: "gpt-4o" });

Image, audio, video, and file parts accept a usable Asset, HTTPS or data URL, Blob, or byte array. AssetRef is a storage reference, not model input. Known unsupported adapter/model combinations fail before provider I/O; custom model identifiers reach the provider's native validation.

The same content works with stream(). Live deltas remain text-only; completion carries the exact ordered assistant content, including reasoning, tool calls, and returned media.

Keep framework edges native

AI SDK ModelMessage and useChat attachments stay native at the route boundary:

import { convertToModelMessages } from "ai";
import { createUIMessageStreamResponse, stream } from "@use-crux/ai";

export async function POST(request: Request) {
  const { messages } = await request.json();
  const result = await stream(chatPrompt, {
    model,
    messages: await convertToModelMessages(messages),
  });
  return createUIMessageStreamResponse(result);
}

Convex Agent likewise owns its file lifecycle, thread history, reloads, tool continuation, and autosave behavior. Crux does not copy those messages or writes.

Ask for a specialized result

Use a specialized operation only when the result shape matters:

import { generateImage, generateSpeech, transcribe } from "@use-crux/openai";

const picture = await generateImage({
  model: "gpt-image-1",
  prompt: "A restrained editorial illustration of a quiet canal",
});

const transcript = await transcribe({
  model: "gpt-4o-mini-transcribe",
  audio: meetingAudio,
});

const narration = await generateSpeech({
  model: "gpt-4o-mini-tts",
  text: transcript.text,
  voice: "alloy",
});

generateImage() returns immediately usable images, transcribe() returns honest measured intervals when available, and generateSpeech() returns one usable audio asset. These are bounded non-streaming operations with cancellation and total/per-attempt timeouts. Their typed extra option is the provider-native escape hatch.

Each bounded call supports canonical Safety. Image and speech accept guardrails and safety; transcription also accepts output-text constraints that run once without provider regeneration. Enforced image strips retain siblings and reset picture.image, while stripping the final image or required speech/transcription audio blocks. An edit mask cannot survive without a reference image. result.safety reports applied canonical decisions; raw and provider-native fields remain unguarded. See the content-primitive matrix.

Use ordinary multimodal generate() when the task is to describe media. Crux does not add a separate description API; specialized operations exist only for guaranteed image, transcript, or audio result shapes.

Unsupported operations are absent from an adapter's type and runtime object. Anthropic, for example, exports none of the three specialized operations. Google transcription is a text-only composition: it returns no word timing or diarization and rejects timing requests instead of inventing measurements.

Persist only when you want reuse

import { inMemoryAssetStore } from "@use-crux/core/storage";

const assets = inMemoryAssetStore();
const stored = await assets.put(picture.image);
const reusableImage = await assets.get(stored.ref);

No model operation accepts an AssetStore or persists implicitly. If put() fails, retry storage without regenerating the media. Treat every AssetRef as a bearer reference owned by its store; namespacing keys is not an authorization boundary.

Derive, retrieve, and evaluate

import { fileSource } from "@use-crux/ingest";

const source = fileSource(storedAudio, {
  namespace: "meetings",
  sourceId: "weekly-sync",
  media: {
    transcribe: ({ audio, abortSignal }) =>
      transcribe({ model: "gpt-4o-mini-transcribe", audio, abortSignal }),
  },
});

await meetingIndex.indexDocuments(source.documents());
const [hit] = await meetingRetriever.retrieve("launch date");
console.log(hit.source.id, hit.source.location);

Image and visual-PDF ingestion use an application-owned media.describe; audio uses media.transcribe; video may use either or both. Derivation becomes ordinary text with page/time attribution. Retrieval returns structured source facts and never hydrates media. Vector metadata stays locator-free.

Eval tasks may accept canonical media parts, but dynamic media whose identity cannot be proven is not reusable. Crux never clones or persists user-owned media merely to create task identity.

Inspect authored and executed media work

Project Index and Catalog show authored generateImage, transcribe, generateSpeech, and ingest operations, including safe modality/options facts and deterministic lints. JavaScript/Oxc and JavaScript/TypeScript-Go produce the same facts.

Observability and Runs show actual media.generate_image, media.transcribe, media.generate_speech, and media.describe attempts, execution mode, latency, safe reports, and derived.from lineage. Local text/transcript capture defaults on; production export defaults off. Neither setting permits bytes, base64/data URLs, AssetRef, provider file IDs, filenames, signed URL credentials, thumbnails, or playback.

Tested adapter matrix

This table is generated from private conformance fixtures. It is documentation, not a runtime capability API; the owning adapter still validates the selected model before I/O.

Errors and remediation

Invalid sources fail with InvalidMediaSourceError; a known unsupported model or portable option fails with UnsupportedCapabilityError; empty or malformed provider output fails validation. Use a supported model, remove the unsupported modality/option, or move provider-native controls into typed extra. Error messages and reports contain safe paths and reason tags, never the original payload.