What happened

• We detected a performance degradation on large portfolios, which caused pages to have high response times and, in some cases, not to load at all.

What we’ve done

• On 2020-05-14, we decided to temporarily revoke access to these resources.

What’s the impact

• 6 users no longer have access to this resource.

What we are doing to help

• We started to add mock data to simulate scenarios with large portfolios, so that we can better understand their behavior and find solutions to improve their performance.
• We are experimenting with various design patterns that allow us to get the necessary data without degrading the performance.
• We will make the appropriate announcement once we enable these features again.

What happened

• We are currently improving our security by enforcing more strict validations. As part of this work, repository protocol input validations were modified to prevent unexpected queries to our API.
• After this change was introduced, we ran a script to assign a protocol to the old repositories that didn’t have one. In the process, we accidentally removed all the repositories from a certain number of projects.

What we’ve done

• We restored the project's database table using a backup from 2020-05-11 at 10:50, one made shortly before we ran the script. With these, we managed to revert all the changes and restore all the deleted repositories. The script restoring all the repositories was run on 2020-05-12 at 15:37.

What’s the impact

• Yesterday, May 11 at 11:10, we ran a script to update some registers in the database. After that, we noticed that a total number of 1050 repositories from 27 projects had been deleted. The time elapsed between the mistake and its fix was 13.5 hours.

What we are doing to help

• From now on, all the scripts, all the scripts that affect production data will be versioned in our repository, through a merge request process.
• We’ve had a very strong backup policy, so we can easily revert our data to a previous state.
• This issue did not affect the security tests nor the database that we use in our testing process. We are currently improving our security by enforcing more strict validations.

Now on ARM, report generation and delivery systems have changed. You can request a report by going to Group -> Vulnerabilities -> Reports.

After a few minutes, you will receive an email like the following:

You can then download the report by clicking the Download button and decrypt it by entering the passphrase, which will be sent to your mobile device through our app.
Download it and register if you haven’t already done so.

What happened

• We are currently updating our API 2 using a completely new backend. As part of this work, and seeking for creating a faster API, a commit 2 was introduced with the goal of improving some technical aspects of data fetching and concurrency handling. Although the change accomplished the desired effect, the current in-memory cache data was not updated to reflect the changes performed.
• This led to unavailability of parts of the application retrieving information using the method introduced by the change.

What we’ve done

• After identifying the problem, the cache was purged to allow filling it with the new data structure

What’s the impact

• The change was deployed to production at 10:28 am of April 23. 12 minutes was the time between the change was deployed to production and the cache was purged.
• During that time, 13 users were affected by the unavailability of retrieving parts of the projects information.

What we are doing to help

• Those kind of changes are rare. However, we identified the pattern of code changes that may affect the in-memory cache structure and defined means to purge the cache when this happens.

What happened

We are currently updating our API using a completely new backend. As part of this work, some bugs have appeared in the migration process. Due to the above, no new resources (repositories, environments, and files) could be created in Integrates between April 6th and 7th.

What we’ve done

When we detect the errors causing the problems, we fixed them in successive Integrates versions.

What’s the impact

• Between 2020-04-06 at 11:28 am and 2020-04-07 at 05:01 pm (13.5 working hours), 18 failed creation attempts were made in Integrates.
• This issue affected 0.8% of our users.

What we are doing to help

We are improving our test cases to prevent further problems like this one.

What happened

We are currently updating our API using a completely new backend. As part of this work, some bugs have appeared in the migration process. Given the above, some findings could not be accepted at Integrates between April 3rd and 6th.

What we’ve done

After we detect the error, we fixed it by correctly validating the user inputs.

What’s the impact

• 6 failed acceptation attempts were made between 2020-04-03 at 03:42 pm and 2020-04-06 at 08:14 am (1.5 working hours)
• This issue affected 0.2% of our users

What we are doing to help

We are improving our test cases to prevent further problems like this one.

What happened

On 2020, from March 26th at 2:00PM to April 2 at 6:47PM (a total of 52 office hours), Integrates suffered of performance issues due to bad table design for the authorization model, enabled after three commits: b44cfa0, 79c48ba, and 6419323:

What we’ve done

We designed a more lightweight database adapter 2, and introduced a cached authorization model in b14adb0 1 and e5d8445 1. Finally, we introduced the best possible table design in a826210, recovering the expected performance levels:

What’s the impact

During 1 week, our users experienced slowness while navigating the application.

What we are doing to prevent this

We added 1200 mock users to the development database in order to early test the application performance.
Also, there is in course a big migration that puts our back-end to work in an asynchronous fashion which highly boosts those operations that have to do with simultaneous database access and retrieval.
We are also always monitoring production systems through a suite of different services, so we can redirect the development towards more performant solutions.

What happened

On April first, 2020, after some weeks migrating the access control system on Integrates from a single-tenant architecture to a multi-tenant one, we received a report with a small list of users that suddenly lost access to the platform.

What we’ve done

We believe the problem was originated due to a slightly deviated migration script. So we ran an integrity check over the database to verify that every single user had the required fields and access attributes set and found a few with some of them missing.
Once we had the list, we manually filled those attributes, reestablishing their access.

What’s the impact

To our knowledge, 0.028% of the users were affected:

• 0.009% were reported by the affected users.
• 0.019% were fixed proactively (before users noticed it).

What happened

On 2020, from March 31th at 3:01PM to April first at 12:25PM, Integrates users tab was not visible to managers due to a bug that was deployed to production in commit da295d8 2. The problem was a miss-leading result given by the async back-end to the front-end, used to resolve if the tab should be shown, or not.

What we’ve done

After we detected the error, we fixed it.

What’s the impact

To our knowledge, 0.025% of the users were affected.
This is because just a few users are project managers.

What we are doing to help

We created an issue (internal-link), in order to add the respective regression tests.

What happened

On March 24th from 7:20PM to 8:20PM, Integrates became inaccessible due to a synchronous data migration of users and roles into a new database table.

What we’ve done

After we detected the error, we immediately reverted that change, deploying a new version of Integrates without the bug. The next day we deployed an asynchronous version of the original change successfully into production

What’s the impact

Approximately 0.8% of our users were affected within the mentioned hour.

What we are doing to help

We are always monitoring our deployments to production, and in this case, we were able to detect the problem and to recover from it in an hour.

Architectural changes are always challenging and even more when they are related to your system’s access control. However, you can have confidence that we are doing our best in doing it as seamlessly as possible