Status

Past incidents

1. Sep, 2024

   1. Database Outage Due to DC Switch FaultEmail DeliveryNotification SenderPing APIDashboard

      Started: 29 Sep, 08:01 UTC

      Duration: 1 hour, 20 minutes

      There's a network fault (a switch fault) at our data-center affecting our primary database. This resulted in a complete outage: dashboard, ping endpoints, notification sender.
      
      We've failed over to a standby database.

      Verifying: 29 Sep, 08:40 UTC

      After fail-over to a standby database systems are back up. 
      We are monitoring the servers, and preparing a new standby.

      Resolved: 29 Sep, 09:21 UTC

      The issue is resolved.

2. Aug, 2024

   All systems operational.

3. Jul, 2024

   1. Stability IssuesDashboard

      Started: 11 Jul, 22:50 UTC

      Duration: 8 hours, 41 minutes

      We're investigating intermittent failing requests on healthchecks.io.

      Identified: 11 Jul, 23:06 UTC

      The HTTP request failures on https://healthchecks.io seem to be a knock-on effects of connectivity issues to our object storage provider.
      
      We have temporarily disabled API calls that retrieve ping bodies from object storage.

      Identified: 12 Jul, 00:20 UTC

      We have verified the failing requests on Healthchecks.io dashboard were indeed related to object storage availability issues. The temporary solution of disabling "get object" API calls has restored the dashboard functionality. Our object storage provider has acknowledged the issue and is working on it.
      
      Currently on Healthchecks.io dashboard, when you view details of a recent ping you will see  a "The request body data is not yet available, please check back later." message. The request bodies are not lost, and will be available for viewing eventually.

      Identified: 12 Jul, 06:18 UTC

      Our object storage provider have implemented a fix. We are currently working through a backlog of not yet uploaded ping bodies.

      Resolved: 12 Jul, 07:32 UTC

      We have fully caught up with ping body backlog, and the service is now operating normally.