Prompt Caching & Context Reuse

Exploits the fact that most prompts share a large stable preamble — schema descriptions, policy text, tool definitions, few-shot examples — by caching its processed form. Usually the single largest cost lever available.

High-volume tasks with a stable prefix. The longer and more repeated the preamble, the larger the saving.

Highly varied one-off prompts with little shared structure, where cache misses dominate.

Cache keys must be tenant-scoped. A cache shared across tenants is a cross-tenant data path, however unlikely the collision.

A perverse incentive against improving prompts, since every edit invalidates the cache and raises cost. Teams stop refining prompts to protect a metric.

A matching-rule explanation task where the schema and rule library form a long fixed preamble and only the specific exception varies.