Silent truncation: the context your model never saw

Somewhere in production right now, a model is answering without its system prompt.

Not because anyone deleted it. The conversation grew, retrieval got generous, someone added a formatting policy last sprint, and the assembled prompt crossed a limit. At that point every layer of the stack has a truncation behavior, and almost all of them are silent. Provider APIs trim input without warning, some from the beginning, some from the middle. Chat frameworks evict oldest messages first. Hand-rolled string assembly just concatenates, and whatever falls past the limit is gone.

No error. No log line. The model answers fluently from whatever survived, and the first symptom is behavioral: the assistant "forgot" a rule, "ignored" an instruction, "lost" its persona.

These bugs are miserable to diagnose for a precise reason: the missing information is missing from the evidence too. The request log shows what was sent. Nothing shows what should have been sent.

Truncation is a policy decision nobody is making

When a prompt doesn't fit, something must be cut. Dropping is unavoidable. But which thing gets cut is a policy decision, and silent truncation means that decision was delegated to accidents: insertion order, message age, provider internals.

Try this on your own stack: if your prompt had to lose 500 tokens right now, what would go? If you can't answer, the answer is "whatever happens to be in the wrong place." One day that will be the compliance disclaimer instead of the third example.

Declaring a representation ladder

A deliberate policy inverts this. Context stays exact and required unless its author explicitly declares a lower-fidelity representation:

import { context, droppable, prefer, prompt } from '@use-crux/core'
import { generate } from '@use-crux/ai'

const rules = context({ id: 'rules', priority: 100, system: CRITICAL_RULES })
const guidelines = context({ id: 'guidelines', priority: 50, system: STYLE_GUIDE })
const compactGuidelines = context({ id: 'guidelines-compact', system: STYLE_SUMMARY })
const examples = context({ id: 'examples', priority: 20, system: FEW_SHOT })

const reply = prompt({
  use: [rules, prefer(guidelines, compactGuidelines), droppable(examples)],
  system: 'You are a support assistant.',
})

const result = await generate(reply, {
  model,
  input,
  inputBudget: { optimizeAt: 1800, max: 2000 },
})

Under pressure, Crux may omit examples or use the compact authored guidelines. rules has no lossy representation, so it remains exact and required regardless of where it appears in use. If no complete legal candidate fits the strict maximum, Crux fails before provider dispatch instead of truncating.

priority orders legal fidelity choices; it does not make a plain context droppable. Capabilities stay attached when a contributor switches to an authored alternative, summary, or exact-recovery reference. An outer droppable() is different: it authorizes removal of the whole contributor, including capabilities owned by it. That distinction keeps prose pressure from silently changing what the model can do.

Excluded, dropped, or stale?

Once dropping is deliberate, it pays to distinguish the ways context can legitimately not reach the model. They have different fixes:

The distinction is not academic, because the mechanisms behave differently. An excluded context's resolver never runs: no wasted work, no tools contributed. Request planning happens after resolution and measures the complete provider request. Representation changes keep owned capabilities sticky; complete omission is possible only when droppable() explicitly authorizes removing the whole contribution.

Collapsing all of these into one undifferentiated "not in the prompt" is why truncation debugging is guesswork. Keeping them separate makes it a lookup.

How do I see what was cut?

Dropping can be acceptable. Silent dropping never is. Every drop should leave a record with a reason, before and after the call.

Before the call, preview the arbitration for any input without executing a provider or preparing new artifacts:

import { preview } from '@use-crux/core'

const prospective = await preview(reply, {
  model,
  input,
  inputBudget: { optimizeAt: 1800, max: 2000 },
})

console.log(prospective.status, prospective.adaptations)

After the call, the executed receipt carries the exact decision:

const receipt = result.steps[0].request
console.log(receipt.adaptations)
const inspection = await receipt.inspect()

The same redacted evidence lands in the trace. In Crux devtools, a turn's Context and Explain views list what was sent, which representation was selected, and every withheld candidate with token counts and reason states.

The debugging session for "the assistant ignored the style guide" becomes: open the turn and find guidelines: authored alternative.

Getting the counts right

Two practical notes that decide whether tight budgets are trustworthy.

Token counting defaults to a chars / 4 estimate. That is fine for coarse budgets and wrong enough to matter for tight ones. Wire a real tokenizer before trusting small margins:

config({
  generation: {
    tokenizer: (text) => encode(text).length,
  },
})

And leave headroom for output. The budget governs input; generation needs its own reserve. A prompt that consumes the whole window leaves the model nothing to answer with, and some providers fail that request in ways that look like model errors rather than budget errors.

For conversation history specifically, arbitrary budget dropping is the wrong tool. Use history.recent() for a causal-group-safe exact suffix, or managed history() to derive a summary while preserving recent turns verbatim. Canonical messages remain unchanged either way. Use representation ladders for composition pressure and history policies for transcript pressure.

Then pin it in CI

"Critical rules always survive" is a testable claim. It is also exactly the kind of invariant that erodes over months of unrelated edits, since every new context block shifts the arbitration. So assert it, against your worst-case inputs:

import { evaluate } from '@use-crux/core/eval'
import { replyTask } from '../src/reply'

export default evaluate({
  id: 'reply-budget-contract',
  task: replyTask,
  cases: worstCaseInputs, // longest histories, fattest retrieval
  expect: (ctx) => {
    ctx.expect.decisionReport.context.toHaveDisposition(
      'context:rules', 'active',
      { reasonCode: 'context.active' },
    )
  },
})

Build worstCaseInputs deliberately: your longest real conversation, the query that retrieves the most chunks, the user with the largest memory. Truncation bugs live at the tail of the input distribution, so testing the median proves nothing. If you import cases from production traces, sort by prompt size and take the top decile.

When next quarter's "add more examples" PR pushes the worst case over budget in a way that would cut the rules, CI fails naming context:rules, instead of a customer finding the persona-less assistant first.

The takeaway

Silent truncation isn't really a context-window problem. Windows keep growing; prompts grow faster. It's an accountability problem: prompts get cut by code that doesn't record what it cut.

Declare the legal representation ladder. Record every adaptation with its reason. Assert the invariants you can't afford to lose. The window can stay finite. The mystery shouldn't.

Whole-request budgets and representation ladders: cruxjs.dev/docs/guides/context-planning. Crux is open source and alpha.