Online Batching with Lookahead: Adversarial Limits and Stochastic Gains

Modern inference systems rely on batching to exploit parallel execution and amortize serial execution overheads, but batching creates a fundamental online scheduling tradeoff: serving requests immediately reduces request queuing delay, while waiting can form larger batches that better utilize parall...

Descripció completa

Dades bibliogràfiques

Autors principals:	Christou, Dimitris, Chen, Jiayi, Sharma, Nihal, Akella, Aditya, Chawla, Shuchi, Shakkottai, Sanjay
Format:	Article
Publicat:	2026
Matèries:	Data Structures and Algorithms
Accés en línia:	https://arxiv.org/abs/2610.16013 http://arxiv.org/abs/2610.16013
Etiquetes:	Afegir etiqueta Sense etiquetes, Sigues el primer a etiquetar aquest registre!