Query engine gateway crashes under unplanned traffic surges
Your team operates a large fleet of Presto clusters spread across multiple data centers, all sitting behind a centralized gateway that routes every SQL query submitted by internal users and automated pipelines. During several incidents over recent months, Presto became completely unavailable to all users — not just a subset. The outages are not correlated with scheduled maintenance windows or cluster-level failures. Individual Presto clusters report healthy uptime during the incidents. The gateway processes, however, stop responding and fail to route any queries, rendering the entire fleet unreachable. Each incident resolves only after manual intervention to restart gateway processes. The pattern does not repeat on a predictable schedule, and post-incident reviews have not surfaced a single misbehaving cluster as the culprit. SREs estimate that each full outage lasts between 15 and 40 minutes and affects all query workloads simultaneously. What's your recommendation to prevent the next unplanned surge from taking down the entire fleet again?