We experienced an issue with existing queries executing in a different fashion following a database version upgrade. These queries exhausted CPU resources on our database, blocking further operations.
Next week a post-event summary will be posted and full root cause analysis will be available through customer's Ada Account Managers.
Posted Apr 15, 2021 - 18:29 EDT
Update
Most of the issues have been taking care of and most services are back up and running. we will continue to monitor the environment and see if there are any more spikes occurring we will work as a team to get a RCA and resolve it so it does not cause any future outages
Posted Apr 15, 2021 - 14:23 EDT
Monitoring
A fix has been implemented and primary functionality has been restored. We are monitoring the fix and working on restoring services to full capacity.
Posted Apr 15, 2021 - 12:51 EDT
Update
we are continuing to work a fix for this issue. we are making progress and we will update as soon as its resolved.
Posted Apr 15, 2021 - 09:09 EDT
Identified
We are experiencing intermittent CPU spikes leading to short periods of downtime on our main production database. We are working with our upstream vendor support to fully investigate and resolve.
Posted Apr 15, 2021 - 07:45 EDT
This incident affected: Ada Dashboard and Ada Web Chat.