Tokenizers

Fuse counts reduced content for planning and output ceilings. Current semantic context uses the configured default counter. Tokenizer selection by model name belongs to the legacy directory-fusion pipeline.

This page is for engineers who want the reported count to match a specific model and maintainers who need the exact resolution rules.

Purpose and Scope

Token counting runs after reduction and secret redaction. The default is o200k_base. The legacy --tokenizer flag accepts a model alias or encoding name; current fuse context, fuse review, and fuse reduce commands do not expose it.

This page documents how a tokenizer value resolves to an encoding. It does not cover the manifest or the reported summary line, which the Core Concepts page describes.

Resolution Rules

The legacy tokenizer value resolves by the following rules. Known aliases are normalized case-insensitively. An unknown value containing an underscore is passed to the tokenizer implementation as supplied.

The Anthropic and Gemini prefixes are matched first, so claude-opus-4 and gemini-1.5-pro select their estimators. A misspelled underscore-containing encoding is not corrected. Any other unknown value falls back to o200k_base.

Estimated Encodings

Anthropic and Google do not publish a local tokenizer vocabulary for their current models, so an exact offline count is not possible; both expose a token-counting API for exact figures. For these families Fuse uses a deterministic estimator: each run of letters and digits costs ceil(length / charsPerToken) tokens and each other non-whitespace character costs one token. The Anthropic estimator uses 3.5 characters per token and the Gemini estimator uses 4, both published rules of thumb for English and code. The result is a stable estimate suitable for budgeting, not an exact count. For an exact offline count, use an OpenAI encoding; for an exact provider-specific count, use that provider's counting API. The estimators are AOT-safe and allocation-light, with no reflection.

Calibrating the Constants

The two constants are the providers' published rules of thumb, not numbers fitted to a measured sample. A calibration harness, tests/benchmarks/harness/calibrate-tokenizers.ps1, measures ground-truth token counts for a held-out C# sample from the pinned corpus through each provider's token-counting API (Anthropic: POST /v1/messages/count_tokens with claude-opus-4-8, needs ANTHROPIC_API_KEY; Gemini: its count-tokens endpoint, needs GEMINI_API_KEY) and reports the chars-per-token band. Run it to check whether the current constants drift on your code, then pin a fitted value if they do. The constants are intentionally left at the published rules of thumb until a measured fit is available, rather than set to an unverified guess. A gated unit test exercises the live comparison when a key is present and skips otherwise.

Choosing an Encoding

For the legacy pipeline, choose the encoding used by the destination model. Current commands do not offer a tokenizer override. Their counts are planning estimates, not a promise that provider-side tokenization will match exactly.

What This Does Not Cover

This page covers token counting and encoding resolution. It does not cover the token budget, the split threshold, or how the manifest reports per-file token costs; those are part of the Emission stage and are documented in the Output Specification.

Next

See Options for current token ceilings and the legacy-option boundary, or Core Concepts for the pipeline.