Статус

Предыдущие инциденты

1. Июл, 2024

   1. Stability IssuesDashboard

      Стартовал: 11 Июл, 22:50 UTC

      Продолжительность: 8 часов, 41 минута

      We're investigating intermittent failing requests on healthchecks.io.

      Определить: 11 Июл, 23:06 UTC

      The HTTP request failures on https://healthchecks.io seem to be a knock-on effects of connectivity issues to our object storage provider.
      
      We have temporarily disabled API calls that retrieve ping bodies from object storage.

      Определить: 12 Июл, 00:20 UTC

      We have verified the failing requests on Healthchecks.io dashboard were indeed related to object storage availability issues. The temporary solution of disabling "get object" API calls has restored the dashboard functionality. Our object storage provider has acknowledged the issue and is working on it.
      
      Currently on Healthchecks.io dashboard, when you view details of a recent ping you will see  a "The request body data is not yet available, please check back later." message. The request bodies are not lost, and will be available for viewing eventually.

      Определить: 12 Июл, 06:18 UTC

      Our object storage provider have implemented a fix. We are currently working through a backlog of not yet uploaded ping bodies.

      Решено: 12 Июл, 07:32 UTC

      We have fully caught up with ping body backlog, and the service is now operating normally.

2. Июн, 2024

   1. Delayed ping body processingNotification SenderDashboard

      Стартовал: 18 Июн, 20:46 UTC

      Продолжительность: 8 часов, 14 минут

      Our object storage provider is currently performing maintenance (scheduled on June 18th, 2024, from 18:00 UTC to 03:00 UTC). Some percentage of object storage API calls are currently failing, which causes the following problems for Healthchecks.io:
      
      * There's a delay in ping body uploads to object storage. When you view details of a recent ping you may see  a "The request body data is not yet available, please check back later." message. The request bodies are not lost, and will be available eventually.
      * Our notification sender process can crash (and silently drop the notification it was about to send) when it hits an unexpected error from object storage. We're currently working on handling these errors. 

      Определить: 18 Июн, 21:11 UTC

      We have deployed a fix for the notification sender process crashing when it hits an unexpected error from object storage.

      Решено: 19 Июн, 05:00 UTC

      Our object storage provider has completed the maintenance. Ping body processing is now back to working normally with no delays.

3. Май, 2024

   1. Database connectivity isssueNotification SenderPing APIDashboard

      Стартовал: 3 Май, 13:24 UTC

      Продолжительность: 1 час, 4 минуты

      We're investigating an issue with our database.

      Расследование: 3 Май, 13:39 UTC

      We have failed over to a standby database instance.

      Проверка: 3 Май, 13:46 UTC

      The systems are back up and working through a backlog of outgoing notifications (caused by pings missed during the downtime).

      Решено: 3 Май, 14:29 UTC

      All systems are operational again. We are currently setting up a new standby database server, to replace the standby that was promoted to primary.
      
      The root cause of the outage was hardware failure: a broken power supply of one of the neighboring servers triggered a fuse in the database server's rack.