On Saturday 20th July, we received reports of profiles not completing their Checker tasks automatically. The engineering team started investigating in business hours on Monday 22nd July and found custom automations responsible for the task were unable to run against some profiles. We identified the root cause on Friday 26th July and rolled out a fix. The automations returned to normal at 6am on Saturday 27th July.
Our internal dashboards showed that the custom automation responsible for completing the certain tasks had no activity since 12 a.m. on July 19. We checked adjacent services and found that the service responsible for sending the task data was stuck and unable to start the custom automation. We identified the cause to be a small number of profiles exceeding size limits. A mitigation was put in place to reduce the size of these profiles. After which, the automations began running again on July 27th.
We truncated redundant information in the large profiles to make them suitable for transmission. Moving forward, we are in the process of rolling out a change to limiting the amount of data sent from large profiles by omitting unused properties in a profile.