Our services are currently down. Looks like there is no UDP connectivity between any of our servers. Investigating.
Verifying:
The services are back up. The outage was caused by a Hetzner-wide network issue: https://status.hetzner.com/incident/5e901309-0873-42af-90c5-84f38304f592
Resolved:
The issue is resolved.
From Hetzner's status page:
The issue was caused due an DDOS attack which was mitigated but blocked too much traffic affecting UDP. Due to the overblocking other services not directly using UDP were influenced too. The issue was solved, we will check further measures to avoid this in the future. All systems are accessible again.
We are investigating network instability between our servers.
Investigating:
The network instability manifested as UDP packet loss between specific server pairs. Currently all servers can contact each other again, and pings are being processed normally. We do not yet know what was causing the packet loss.
Verifying:
Hetzner has posted an incident report about a core router fault. The incident start time (23:05 UTC) lines up precisely with the moment we started seeing packet loss, so this is most likely the cause of the packet loss.
We will be upgrading HAProxy on our load balancer servers on Tuesday, January 6, 2026. Planned changes:
* Upgrade HAProxy to version 3.2 * Update the list of accepted SSL/TLS ciphers from a hand-picked list to HAProxy defaults * Later in the week, gradually enable HTTP/3.
These changes should not affect the vast majority clients, but could in theory cause connection issues for very old legacy clients. If you notice persistent ping request failures starting from January 6, please let us know.