Multimodal content

import {
  contentText,
  hasMediaParts,
  messageText,
  textPart,
} from "@use-crux/core";
import type { ContentPart, MediaSource, MessageContent } from "@use-crux/core";

For task-oriented examples, start with Media inputs. Specialized completed and streaming operations have separate media-operation and media-streaming references.

MessageContent

Canonical generation messages use:

type MessageContent = string | readonly ContentPart[];

ContentPart is a closed model-visible union:

Assistant content may additionally contain ordered ReasoningPart and ToolCallPart values. System, user, and tool messages cannot. On a generation result, content is authoritative and text is its text-part projection.

MediaSource

MediaSource accepts:

• Asset;
• HTTPS or data URL string;
• URL;
• Uint8Array;
• ArrayBuffer;
• Blob.

AssetRef is intentionally excluded. Hydrate a reference through its owning AssetStore before passing the returned StoredAsset to a model.

Model calls normalize and validate sources before provider I/O. They do not download arbitrary sources for capability discovery or persist media.

Construct parts

textPart(text)

Create a canonical text part:

const part = textPart("Summarize this image.");

Media parts are plain typed objects:

const image: ContentPart = {
  type: "image",
  source: pngBytes,
  mediaType: "image/png",
};

const file: ContentPart = {
  type: "file",
  source: "https://example.com/report.pdf",
  mediaType: "application/pdf",
  filename: "report.pdf",
};

const audio: ContentPart = {
  type: "audio",
  source: wavBytes,
  mediaType: "audio/wav",
};

Project text

Projection helpers are for model-facing semantic text. They are not logging redactors.

contentText(content)

Project a MessageContent value into bounded text:

const projected = contentText([
  textPart("Inspect this."),
  { type: "image", source: pngBytes, mediaType: "image/png" },
]);

Text parts are emitted verbatim. Other parts become semantic descriptors that may include a locator or filename.

messageText(message)

Project message.content:

const projected = messageText(message);

Use this where a widened message must become semantic text, such as guardrails, compaction, memory capture, cache query text, or a custom provider fallback. Observability applies a stricter sanitizer.

hasMediaParts(content)

Return true when a content array contains a non-text part:

if (hasMediaParts(message.content)) {
  // apply media-aware handling
}

Errors

Malformed media throws InvalidMediaSourceError. Valid media that the selected adapter or known model cannot send throws UnsupportedCapabilityError before provider I/O. Unknown custom models reach the provider's native validation.

Privacy

Observability never uses contentText() as its media redactor. At every capture level it excludes bytes, base64/data URLs, bearer refs, provider file IDs, filenames, hashes, signed URL details, and native media payloads.

Convex mirror

@use-crux/convex re-exports ContentPart, MessageContent, textPart, contentText, messageText, and hasMediaParts from its root.