cp

A new engineering writeup describes routing large language model traffic across multiple inference providers using congestion-control principles borrowed from TCP. The approach adaptively shifts requests toward providers with lower latency and fewer errors, easing off when a provider slows down. Hacker News readers are engaging with the technical details, reflecting wider industry interest in making multi-provider AI setups more reliable.

A talk on Homa, a transport protocol designed to replace TCP in modern data centres, is drawing attention. Homa is a receiver-driven protocol aimed at cutting tail latency and improving performance in machine learning clusters, where traffic patterns differ sharply from those TCP was built for.