Service Outage

Incident Report for 2talk

Postmortem

Description: Network Outage

Date: 29 Oct: 13:30hrs - 14:30 hrs

Severity: Complete outage

We use LXC containers to provide high availability (HA) redundancy with all core assets within our business. As part of our normal growth our DevOps team is continuing to add new server infrastructure to our core clusters.
This outage was caused by the additional of a new physical node into our LXC Container cluster. We believe the fault was a software issue within the distributed filesystem required for HA and redundancy between containers.
While this shouldn’t have happened, we will continue investigating the underlying cause; however, the simple resolution is to avoid adding new nodes (or servers) during operational hours.

Mike Johnstone | CTO

Posted Oct 29, 2021 - 15:17 PDT

Resolved

All access has now been restored to normal.

Posted Oct 29, 2021 - 14:46 PDT

Update

Services are being restored presently. We apologize for this disruption.

Posted Oct 29, 2021 - 14:29 PDT

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Oct 29, 2021 - 13:53 PDT

Update

Our team expects to have services restored within the next 15 minutes. Please bear with us during this time.

Posted Oct 29, 2021 - 13:50 PDT

Update

Our team has identified the issue and is working to resolve this.

Posted Oct 29, 2021 - 13:45 PDT

Update

We have identified the issue and are working on a resolution within the following 10 mins

Posted Oct 29, 2021 - 13:40 PDT

Investigating

Customers are experiencing a service disruption right now. Our engineers are investigating this and will have all services fully restored as our highest priority.

Posted Oct 29, 2021 - 13:37 PDT

This incident affected: 2talk Calling Platform (Network_, 2talk Cloud PBX, and 2vFax.