Problem
Throughput-Latency Interference
Serving strategies that maximize raw throughput can violate first-token or completion-latency objectives.
Problem
Serving strategies that maximize raw throughput can violate first-token or completion-latency objectives.