Summary

On Thursday 25th May at around 06:13 BST, we received an alert that some requests to the PassFort platform were failing. After completing an investigation and remediation work, full service was restored around 09:00 BST.

Analysis

During the incident, some requests to recently deployed resources in the PassFort platform were unsuccessful due to internal network instability. Requests to less-recently deployed resources were successful.

This internal network instability was itself caused by a subset of overnight batch processing jobs being incorrectly scheduled to hardware reserved for live traffic.

Remediation

• The configuration of the overnight batch jobs has been updated to better isolate them from adjacent resources
• We have re-prioritised a project to replace the currently deployed network technology