Status

Wcześniejsze incydenty

1. Paź, 2021

   1. Ping processing issuePing API

      Rozpoczął się: 11 Paź, 16:08 UTC

      Czas trwania: 34 minuty

      At 16:08 UTC, one of the app servers encountered a brief connectivity issue to the database. To recover, a worker process had to be restarted. At the time it had ~4200 incoming pings queued in the memory, and these pings were lost during restart. The app server is now back to normal.

      Post-mortem

      Some more details on the issue.
      
      Healthchecks Ping API has a mitigation for times when the database is briefly unavailable (network outage, database restart, database failover). The mitigation is to queue incoming pings in memory, and keep retrying to connect to the database. Thanks to this mechanism, workers can usually recover from hiccups automatically – they reconnect to the database and work through the queue.
      
      Healthchecks Ping API used to respond with "200 OK" to client HTTP requests as soon a the request was received, before it was recorded to the database. The "200 OK" response effectively meant "request received, understood and queued".
      
      After deploying a recent new feature, slug URLs, this changed. Ping API now returns a response to the client only after the ping is recorded to the database. This means that during a database outage the number of open and hanging client connections can grow quickly. If the number of open connections reaches the "Max open files" limit, the worker process cannot recover any more – it cannot create another new connection to reconnect to the database. This is what happened during this brief outage: there was a network glitch, the worker exhausted the "Max open files" limit (4096) and got stuck in the endless loop trying to reconnect to the database. The immediate "fix" was to restart the worker process. And the proper fix was to increase the open files limit from 4096 to a much higher value – this is done now.
      
      If you received a false alert from Healthchecks.io about one of your checks being down around 16:10 UTC – this might be why. Apologies!

      Rozwiązany: 11 Paź, 16:42 UTC

      The issue is resolved.

2. Wrz, 2021

   Wszystkie systemy działają.

3. Sie, 2021

   1. Network connectivity problemsNotification SenderPing APIDashboard

      Rozpoczął się: 24 Sie, 11:23 UTC

      Czas trwania: 1 godzina, 14 minut

      We had a network issue between August 24, 11:23 UTC and 11:28 UTC. One of the app servers lost connectivity to the database, causing ping processing delay and dashboard outage for 33% of the traffic.

      Rozwiązany: 24 Sie, 12:37 UTC

      The issue is resolved.