Delay in automation
Incident Report for Passfort
Postmortem

At 16:47 on 20th March, the internal webhooks queue for ADQ Sapiens stalled. This was due to out-of-memory errors caused by large payload size. The requested memory for the Sapiens ingestors was increased, and the backlog of messages began processing on 22nd March at 16:23. As there was a large backlog, it took several days to finish processing, and was completed at 24th March 11:00. In response to this incident, we have increased memory allocation for the services involved, and are working to make our webhooks service more robust.

Posted Apr 09, 2024 - 09:55 BST

Resolved
Our engineers have advised the issue has been resolved and the backlog has been cleared.
Posted Mar 25, 2024 - 09:49 GMT
Update
We are continuing to monitor and process the backlog caused by this issue.
Posted Mar 22, 2024 - 10:45 GMT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Mar 21, 2024 - 22:57 GMT
Identified
The issue has been identified and a fix is being implemented.
Posted Mar 21, 2024 - 21:19 GMT
This incident affected: PassFort Environments (🇦🇪 UAE - identity.uae.passfort.com).