Status

Past incidents

1. Jan, 2022

   All systems operational.

2. Dec, 2021

   1. Degraded performance due to database overloadNotification SenderPing APIDashboard

      Started: 16 Dec, 05:30 UTC

      Duration: 14 hours, 35 minutes

      At 5:30 UTC, our primary database became overloaded, this caused slow HTTP response times, and notification delays. 

      Investigating: 16 Dec, 06:35 UTC

      The database is back to normal currently. The overload seems to have been triggered by a scheduled nightly database backup, but still investigating.

      Resolved: 16 Dec, 20:05 UTC

      The issue is resolved.
   2. Network connectivity problemsPing APIDashboard

      Started: 15 Dec, 20:28 UTC

      Duration: 11 hours, 40 minutes

      Our hosting provider is currently performing a backbone connection maintenance. This is causing intermittent latency spikes and connection timeouts.
      
      https://status.hetzner.com/incident/b90d7054-17af-4eae-a861-7ea9bfc489db

      Identified: 15 Dec, 21:31 UTC

      The network latency seems to have stabilized but is still higher than usual, and the network maintenance is still underway. The posted maintenance window is 2021-12-15 20:00 UTC+0 – 2021-12-16 15:00 UTC+0.

      Resolved: 16 Dec, 08:08 UTC

      The hosting provider has completed the scheduled maintenance (https://status.hetzner.com/incident/b90d7054-17af-4eae-a861-7ea9bfc489db).
      
      The network latency between Frankfurt and Falkenstein is back to normal levels (5ms ICMP ping, 10ms for a complete HTTP request).

3. Nov, 2021

   1. Network issuesNotification SenderPing APIDashboard

      Started: 5 Nov, 12:27 UTC

      Duration: 42 minutes

      We are experiencing network connectivity issues – several brief periods of 100% packet loss between specific servers.
      
      There's a network incident at our hosting provider, likely related: https://status.hetzner.com/incident/d1e59188-9843-4801-a01e-a7ba5fd75940

      Resolved: 5 Nov, 13:09 UTC

      The issue is resolved.