What happened

On 2020, from March 31th at 3:01PM to April first at 12:25PM, Integrates users tab was not visible to managers due to a bug that was deployed to production in commit da295d8 2. The problem was a miss-leading result given by the async back-end to the front-end, used to resolve if the tab should be shown, or not.

What we’ve done

After we detected the error, we fixed it.

What’s the impact

To our knowledge, 0.025% of the users were affected.
This is because just a few users are project managers.

What we are doing to help

We created an issue (internal-link), in order to add the respective regression tests.

What happened

On March 24th from 7:20PM to 8:20PM, Integrates became inaccessible due to a synchronous data migration of users and roles into a new database table.

What we’ve done

After we detected the error, we immediately reverted that change, deploying a new version of Integrates without the bug. The next day we deployed an asynchronous version of the original change successfully into production

What’s the impact

Approximately 0.8% of our users were affected within the mentioned hour.

What we are doing to help

We are always monitoring our deployments to production, and in this case, we were able to detect the problem and to recover from it in an hour.

Architectural changes are always challenging and even more when they are related to your system’s access control. However, you can have confidence that we are doing our best in doing it as seamlessly as possible

What happened

• Last Monday (March 16th) there was an incident in which mails about events pending to solve were sent to projects without unsolved events.
• The body of the mail indicated that the project had 0 events that were restricting the execution of the project.
• The incident was caused by an error in the interpretation of a zero (0) value, in which it was taken as a string instead of a number, causing the automatic script not to ignore the projects with this situation.

What we’ve done

• That mail is sent every week on Mondays, we have now fixed it, and next week the mail will be sent correctly.

What’s the impact

• 120 mails were sent to 40 users in 16 projects that have no pending to solve events.

What we are doing to help

• We are checking all the possible errors caused by the typing refactor in which we are working.

What happened

On March 13th from 5:34PM to 6:05PM, Integrates became inaccessible due to a change 2 in the communication with the new API in which we are working during the last weeks.

What we’ve done

After we detect the error, we immediately revert that change, deploying a new version of Integrates without the bug.

What’s the impact

Approximately 12 login attempts were unsuccessful within the mentioned 31 minutes.

What we are doing to help

We are always monitoring our deployments to production, and in this case, we were able to detect the problem and to recover from it in half an hour.

Now on ARM, it is possible to delete your own projects. To do so, you have to go to the Settings tab and scroll down to the “Delete Project” button:

You will have to type the project name to confirm that you really want to delete the project. After that, you will be redirected to the dashboard, where you are going to see that the project is still available, but if you access it, only information about the deletion will be shown:

Some clarifications:

• The project will be deleted after 30 days of the requested date.
• You need Manager level access to the project to perform this action.
• After you request the deletion of the project, only you can cancel it at any time before the deletion date.
• The project will only appear in the dashboard to the person who requests the deletion.
• No notifications will be sent for a pending-to-delete project.
• In the deletion process, we mask all sensitive information that can connect the vulnerabilities with our customers, and we only keep generic data for analytical purposes. This data gives us the possibility to generate valuable reports like this year’s Annual Report.

What happened

• Along March, we deployed a lot of changes to our continuous integration and delivery system with the aim of improving environments reproducibility.
• On March 9 the schedule in charge of deploying a new integrates version failed.
• Since we deploy integrates from this system in an automated way with the purpose of rotating our AWS keys and the scheduled deployment failed, the AWS Keys in the previous version of Integrates (which customers were accessing on March 8) expired.
• Due to that, our back-end was not able to communicate to AWS services like DynamoDB (the database) and S3 (finding evidences) among others, causing a downtime in the service.

What we’ve done

• We detected the issue on March 9th and deployed a new version of Integrates with fresh AWS keys manually.

What’s the impact

• Unavailability of users’ and findings’ data between 4am and 7am (Colombian Time).

What we are doing to help

• We fixed and make sure the continuous delivery system works as expected. We are investigating ways to avoid future occurrences of expired AWS keys in the system.

What happened

On March 4th from 08:38AM to 8:50AM, Integrates became inaccessible due to an overload in the Kubernetes cluster, which was the result of new experimental ephemeral environments not being shut down.

What we’ve done

When the cause of the problem was found, we restarted the cluster’s overloaded nodes,
added extra nodes to increase computing capabilities and programmed a function to remove experimental ephemeral environments.

What’s the impact

Users were not able to access Integrates within the mentioned 12 minutes.

What we are doing to help

We restarted the overloaded nodes, increased the cluster size and programmed a function for stopping experimental ephemeral environments.

Now, on Integrates, it is possible to request the verification by vulnerabilities rather than by findings.

• From this day, when you press the “Request verification” button, you will see the list of vulnerabilities, and you will be able to choose which ones you want our hackers to verify.
• After you select the vulnerabilities, you have to press the “Request verification” button below the vulnerabilities and you will be prompted to enter a justification.
• The “Verification” column in the vulnerabilities table indicates the verification status.

You may find a more precise explanation of the new process in the following GIF:

• The comment places the vulnerabilities that are going to be verified before the justification.

• With this functionality, our analysts are able to perform a partial verification, so if there is an event that affects the verification of some of the vulnerabilities for which it was requested, they can do so only for those they can access.

What happened

• On February 20th and 24th we deployed a couple of changes to our backend with the aim of improving Integrates’s performance on the database side.
• Since we have infrastructure as code, we use the Terraform language to make changes to the database. Unfortunately, this tool has a bug related to the database attributes that we added in the changes mentioned above.
• Due to that bug, some database keys were re-created after every deployment, and that ended up causing some delays in the responses to our backend.

What we’ve done

• We detected some delay issues on February 24th and began to investigate the causes. We found the problem on February 25th and finally applied the recommended fix 1 on February 26th.

What’s the impact

• Approximately 36 errors were generated due to this issue from February 24th to 26th, causing delays in the responses and, in some cases, unavailability in the content of users’ and findings’ tabs.

What we are doing to help

• We are now aware of the presence of this bug and will remain vigilant so that we can avoid these issues whenever we make any changes to table keys.

What happened

• Last week’s Monday and Tuesday (February 17th and 18th) and this week’s Monday (February 24th) in the afternoon we released new versions of Integrates that caused a failure in the continuous deployment of Integrates.
• We have a daily deployment that rotates some security keys that last only for 24 hours to preserve the security of our customers.
• In those previously mentioned days, the rotation jobs failed, so the keys used by the app were invalid, thus the access to all the projects was forbidden.

What we’ve done

• On all three days we manually triggered the rotation job early in the morning and, after that, we fixed the code that caused the failure on the pipelines.

What’s the impact

• The issue lasted approximately 5 hours (4:10-9:21) on February 18th, 3 hours (4:10-7:04) on February 19th and 3 hours (4:10-7:02) on February 25th. However, access attempts only occurred from 6:50 to 9:20 on the first day, from 6:44 to 7:02 on the second one and 6:26 to 6:58 on the third one, resulting in a total real affectation of around 3.5 hours.
• Approximately 45 of our users were unable to access their projects on February 18th, 19th and 25th in the morning.

What we are doing to help

• We are developing an automatic test that checks the access to the projects and that notifies us by SMS and mail whenever it fails.
• We are increasing the availability window of the old keys to two days, in this way, if a nightly pipeline fails, we have 24 hours to fix the errors without causing availability affectation to our customers.
• With this announcement, customers are being notified that this was an internal error caused by two failures on deployment pipelines and of what we did to fix it.