Summary

All times are in UTC.

At about 10:35 on 2021-09-14 customers started reporting increased delays in application processing.

An investigation was started immediately. At 11:30 we received an alert from our monitoring system informing us of delays in risk processing.

PassFort engineers determined the cause of the issue was an unexpectedly large number of risk jobs, coupled with an unoptimised query, that had led to a rising backlog of jobs in our Risk service.

At about 13:00 a fix was deployed to speed up risk processing. Risk throughput was greatly increased after this point and the backlog was cleared within an hour.

PassFort engineers performed a sanity check on other potentially affected services and noticed an increased backlog in our Search service, caused by a similar issue. This meant new applications could not be found via the portal. At about 16:15 a fix was deployed, and by 16:30 the backlog had been cleared.

Analysis

The rising backlog of risk jobs was due to an unusually large number of applications being created on our system on that day.

Most of our services were able to scale without any issues. The Risk and Search services were using unoptimised queries when accessing the database, which slowed down job processing.

Actions

1. We have updated our services to ensure frequent database queries use appropriate indexes
2. We are updating our dashboards to have better visibility on increasing job backlogs across our services