4 years ago
About response time issues on 2020-02
What happened
- On February 20th and 24th we deployed a couple of changes to our backend with the aim of improving Integrates’s performance on the database side.
- Since we have infrastructure as code, we use the Terraform language to make changes to the database. Unfortunately, this tool has a bug related to the database attributes that we added in the changes mentioned above.
- Due to that bug, some database keys were re-created after every deployment, and that ended up causing some delays in the responses to our backend.
What we’ve done
- We detected some delay issues on February 24th and began to investigate the causes. We found the problem on February 25th and finally applied the recommended fix 1 on February 26th.
What’s the impact
- Approximately 36 errors were generated due to this issue from February 24th to 26th, causing delays in the responses and, in some cases, unavailability in the content of users’ and findings’ tabs.
What we are doing to help
- We are now aware of the presence of this bug and will remain vigilant so that we can avoid these issues whenever we make any changes to table keys.