What the model actually needs#
Most enterprise AI tasks are not about the sensitive value itself — they are about the structure around it. Summarize the complaint. Draft the reply. Decide which team should own the ticket. For those tasks the model needs to know that the caller and the account holder are the same person, that the account is an account, and that the third mention of the customer is still the same customer. It rarely needs the actual digits of the phone number.
Typed, stable tokens carry exactly that structure. PERSON_81af tells the model it is dealing with a person, and every recurrence of the same person carries the same token — through a long document, a multi-turn conversation, or a streamed response. The model can resolve references and reason about relationships the way it would over real values.
Why undifferentiated masking breaks this#
Redaction collapses every sensitive value into the same opaque marker. The model loses two things at once: the category (is this a person, a place, an account?) and the identity (are these two markers the same entity?). On any prompt with more than one sensitive value, the reasoning degrades quickly — a summary of a complaint with three indistinguishable blanks cannot say who did what.
Token scope is the tuning knob here: within a conversation, a document, or a wider configured scope, the same value maps to the same token, which is what keeps cross-references intact. Tokenization covers scope in detail.
An honest boundary#
Context preservation is about structure, not magic. The model still loses the value itself, so operations that need the real characters — arithmetic on a tokenized amount, validating a checksum, spell-checking a name — degrade, as documented in Tokenization. The claim worth making is precise: this preserves more useful structure than deletion or undifferentiated masking, and for the reasoning tasks enterprises actually run, that structure is usually what quality depends on.