Now on ARM, report generation and delivery systems have changed. You can request a report by going to Group -> Vulnerabilities -> Reports.

After a few minutes, you will receive an email like the following:

You can then download the report by clicking the Download button and decrypt it by entering the passphrase, which will be sent to your mobile device through our app.
Download it and register if you haven’t already done so.

What happened

• We are currently updating our API 2 using a completely new backend. As part of this work, and seeking for creating a faster API, a commit 2 was introduced with the goal of improving some technical aspects of data fetching and concurrency handling. Although the change accomplished the desired effect, the current in-memory cache data was not updated to reflect the changes performed.
• This led to unavailability of parts of the application retrieving information using the method introduced by the change.

What we’ve done

• After identifying the problem, the cache was purged to allow filling it with the new data structure

What’s the impact

• The change was deployed to production at 10:28 am of April 23. 12 minutes was the time between the change was deployed to production and the cache was purged.
• During that time, 13 users were affected by the unavailability of retrieving parts of the projects information.

What we are doing to help

• Those kind of changes are rare. However, we identified the pattern of code changes that may affect the in-memory cache structure and defined means to purge the cache when this happens.

What happened

We are currently updating our API using a completely new backend. As part of this work, some bugs have appeared in the migration process. Due to the above, no new resources (repositories, environments, and files) could be created in Integrates between April 6th and 7th.

What we’ve done

When we detect the errors causing the problems, we fixed them in successive Integrates versions.

What’s the impact

• Between 2020-04-06 at 11:28 am and 2020-04-07 at 05:01 pm (13.5 working hours), 18 failed creation attempts were made in Integrates.
• This issue affected 0.8% of our users.

What we are doing to help

We are improving our test cases to prevent further problems like this one.

What happened

We are currently updating our API using a completely new backend. As part of this work, some bugs have appeared in the migration process. Given the above, some findings could not be accepted at Integrates between April 3rd and 6th.

What we’ve done

After we detect the error, we fixed it by correctly validating the user inputs.

What’s the impact

• 6 failed acceptation attempts were made between 2020-04-03 at 03:42 pm and 2020-04-06 at 08:14 am (1.5 working hours)
• This issue affected 0.2% of our users

What we are doing to help

We are improving our test cases to prevent further problems like this one.

What happened

On 2020, from March 26th at 2:00PM to April 2 at 6:47PM (a total of 52 office hours), Integrates suffered of performance issues due to bad table design for the authorization model, enabled after three commits: b44cfa0, 79c48ba, and 6419323:

What we’ve done

We designed a more lightweight database adapter 2, and introduced a cached authorization model in b14adb0 1 and e5d8445 1. Finally, we introduced the best possible table design in a826210, recovering the expected performance levels:

What’s the impact

During 1 week, our users experienced slowness while navigating the application.

What we are doing to prevent this

We added 1200 mock users to the development database in order to early test the application performance.
Also, there is in course a big migration that puts our back-end to work in an asynchronous fashion which highly boosts those operations that have to do with simultaneous database access and retrieval.
We are also always monitoring production systems through a suite of different services, so we can redirect the development towards more performant solutions.

What happened

On April first, 2020, after some weeks migrating the access control system on Integrates from a single-tenant architecture to a multi-tenant one, we received a report with a small list of users that suddenly lost access to the platform.

What we’ve done

We believe the problem was originated due to a slightly deviated migration script. So we ran an integrity check over the database to verify that every single user had the required fields and access attributes set and found a few with some of them missing.
Once we had the list, we manually filled those attributes, reestablishing their access.

What’s the impact

To our knowledge, 0.028% of the users were affected:

• 0.009% were reported by the affected users.
• 0.019% were fixed proactively (before users noticed it).

What happened

On 2020, from March 31th at 3:01PM to April first at 12:25PM, Integrates users tab was not visible to managers due to a bug that was deployed to production in commit da295d8 2. The problem was a miss-leading result given by the async back-end to the front-end, used to resolve if the tab should be shown, or not.

What we’ve done

After we detected the error, we fixed it.

What’s the impact

To our knowledge, 0.025% of the users were affected.
This is because just a few users are project managers.

What we are doing to help

We created an issue (internal-link), in order to add the respective regression tests.

What happened

On March 24th from 7:20PM to 8:20PM, Integrates became inaccessible due to a synchronous data migration of users and roles into a new database table.

What we’ve done

After we detected the error, we immediately reverted that change, deploying a new version of Integrates without the bug. The next day we deployed an asynchronous version of the original change successfully into production

What’s the impact

Approximately 0.8% of our users were affected within the mentioned hour.

What we are doing to help

We are always monitoring our deployments to production, and in this case, we were able to detect the problem and to recover from it in an hour.

Architectural changes are always challenging and even more when they are related to your system’s access control. However, you can have confidence that we are doing our best in doing it as seamlessly as possible

What happened

• Last Monday (March 16th) there was an incident in which mails about events pending to solve were sent to projects without unsolved events.
• The body of the mail indicated that the project had 0 events that were restricting the execution of the project.
• The incident was caused by an error in the interpretation of a zero (0) value, in which it was taken as a string instead of a number, causing the automatic script not to ignore the projects with this situation.

What we’ve done

• That mail is sent every week on Mondays, we have now fixed it, and next week the mail will be sent correctly.

What’s the impact

• 120 mails were sent to 40 users in 16 projects that have no pending to solve events.

What we are doing to help

• We are checking all the possible errors caused by the typing refactor in which we are working.

What happened

On March 13th from 5:34PM to 6:05PM, Integrates became inaccessible due to a change 2 in the communication with the new API in which we are working during the last weeks.

What we’ve done

After we detect the error, we immediately revert that change, deploying a new version of Integrates without the bug.

What’s the impact

Approximately 12 login attempts were unsuccessful within the mentioned 31 minutes.

What we are doing to help

We are always monitoring our deployments to production, and in this case, we were able to detect the problem and to recover from it in half an hour.