E-mail
Slack
Microsoft Teams
Telegram
Discord
Feed
Calendar
Welcome to the Healthchecks.io status page. If there are interruptions to service, we will post a report here, and on our Mastodon account.
Started:
Duration:
Here's a quick recap of this outage.
Yesterday, Hetzner datacenters in Falkenstein were hit with by a large DDOS attack. As a mitigation, Hetzner throttled UDP traffic on ports 9000 and above.
Healthchecks.io uses Wireguard for private communication between servers (load balancers to web servers, web servers to database servers). Wireguard works over UDP, and, after the throttling started, the available bandwidth between servers dropped to below 1Mbit/s.
After figuring out what had happened, I updated Wireguard configuration to use a port number below 9000. After deploying the change, Healthchecks resumed normal operation.
The outage lasted almost 2 hours. During the outage, the ping API was accepting and processing some but not all pings. The web UI and the notification sender was completely non-operational. When normal operation resumed, Healthchecks sent out a wave of false alerts due to pings that were not received on time.
This was an unfortunate event, I apologize for the trouble caused by failing pings, non-operational management API, and the eventual false alerts. Still, there are also several positive aspects, in the "it could have been worse" sense, I would like to acknowledge:
Yesterday, Hetzner datacenters in Falkenstein were hit with by a large DDOS attack. As a mitigation, Hetzner throttled UDP traffic on ports 9000 and above.
Healthchecks.io uses Wireguard for private communication between servers (load balancers to web servers, web servers to database servers). Wireguard works over UDP, and, after the throttling started, the available bandwidth between servers dropped to below 1Mbit/s.
After figuring out what had happened, I updated Wireguard configuration to use a port number below 9000. After deploying the change, Healthchecks resumed normal operation.
The outage lasted almost 2 hours. During the outage, the ping API was accepting and processing some but not all pings. The web UI and the notification sender was completely non-operational. When normal operation resumed, Healthchecks sent out a wave of false alerts due to pings that were not received on time.
This was an unfortunate event, I apologize for the trouble caused by failing pings, non-operational management API, and the eventual false alerts. Still, there are also several positive aspects, in the "it could have been worse" sense, I would like to acknowledge:
- TCP was still working. I could access the servers over SSH the whole time, so I had at least some control over the situation.
- The Wireguard port change worked as a workaround. Without it, the outage would have continued several more hours.
- The primary database server got a long overdue reboot, and is now running a newer kernel.
- When the problem hit, I was at home, awake, and able to respond immediately.
PS. If you notice any lingering issues, have any suggestions or questions, please let me know at contact@healthchecks.io. Thank you!
–Pēteris
–Pēteris