KPIsShared across all subject areas

p95 inference latency

The response time 95% of model calls fall under.

Formula

95th percentile of end-to-end model call duration, including gateway overhead

KPI cautions1
  • Measuring only the provider's reported generation time, excluding gateway, retrieval and queueing — which is most of what the user experiences.
KPI lifecycle stages1