What happened

• We are currently improving our security by enforcing our session user token validations.
• After this change was introduced, API tokens were accidentally validated when they shouldn’t.

What we’ve done

• We noticed the issue today Jun 23 at 14:37 PM, and committed the fix at 15:40 PM.

What’s the impact

• Today, Jun 23, we introduced the token validation change at 13:40 PM and some users weren’t able to successfully query our API.

What we are doing to help

• We are improving our tests in order to cover all the scenarios related with that token functionality.

What happened

On June 3rd, from 6:30 PM to 7:00 PM, users experienced a performance degradation in the platform, making the browsing really slow. This occurred because one of our analysts was performing a DDoS attack on the site. At the same time, there was a failed deployment in production that halved the number of machines processing requests; half of them had the previous version of the app and were responding correctly, while the other half were stuck in a loop trying to deploy the new version which had issues. This attenuated the effect of the DDoS.

What we’ve done

After we detected the error, we immediately reverted that change, deploying a new version of Integrates without the bug, so the whole fleet was available.

What’s the impact

The incident occurred outside of working hours, so only 3 users were affected, and they experienced a really slow platform, with group information taking too long to load or not loading at all.

What we are doing to help

We are constantly performing security tests over our own platform to discover these kinds of flaws. Right now, we are working on concurrency improvements to achieve a better performance. Also, this vulnerability was reported to our developer team and will be addressed shortly

What happened

• We detected a performance degradation on large portfolios, which caused pages to have high response times and, in some cases, not to load at all.

What we’ve done

• On 2020-05-14, we decided to temporarily revoke access to these resources.

What’s the impact

• 6 users no longer have access to this resource.

What we are doing to help

• We started to add mock data to simulate scenarios with large portfolios, so that we can better understand their behavior and find solutions to improve their performance.
• We are experimenting with various design patterns that allow us to get the necessary data without degrading the performance.
• We will make the appropriate announcement once we enable these features again.

What happened

• We are currently improving our security by enforcing more strict validations. As part of this work, repository protocol input validations were modified to prevent unexpected queries to our API.
• After this change was introduced, we ran a script to assign a protocol to the old repositories that didn’t have one. In the process, we accidentally removed all the repositories from a certain number of projects.

What we’ve done

• We restored the project's database table using a backup from 2020-05-11 at 10:50, one made shortly before we ran the script. With these, we managed to revert all the changes and restore all the deleted repositories. The script restoring all the repositories was run on 2020-05-12 at 15:37.

What’s the impact

• Yesterday, May 11 at 11:10, we ran a script to update some registers in the database. After that, we noticed that a total number of 1050 repositories from 27 projects had been deleted. The time elapsed between the mistake and its fix was 13.5 hours.

What we are doing to help

• From now on, all the scripts, all the scripts that affect production data will be versioned in our repository, through a merge request process.
• We’ve had a very strong backup policy, so we can easily revert our data to a previous state.
• This issue did not affect the security tests nor the database that we use in our testing process. We are currently improving our security by enforcing more strict validations.

Now on ARM, report generation and delivery systems have changed. You can request a report by going to Group -> Vulnerabilities -> Reports.

After a few minutes, you will receive an email like the following:

You can then download the report by clicking the Download button and decrypt it by entering the passphrase, which will be sent to your mobile device through our app.
Download it and register if you haven’t already done so.

What happened

• We are currently updating our API 2 using a completely new backend. As part of this work, and seeking for creating a faster API, a commit 2 was introduced with the goal of improving some technical aspects of data fetching and concurrency handling. Although the change accomplished the desired effect, the current in-memory cache data was not updated to reflect the changes performed.
• This led to unavailability of parts of the application retrieving information using the method introduced by the change.

What we’ve done

• After identifying the problem, the cache was purged to allow filling it with the new data structure

What’s the impact

• The change was deployed to production at 10:28 am of April 23. 12 minutes was the time between the change was deployed to production and the cache was purged.
• During that time, 13 users were affected by the unavailability of retrieving parts of the projects information.

What we are doing to help

• Those kind of changes are rare. However, we identified the pattern of code changes that may affect the in-memory cache structure and defined means to purge the cache when this happens.

What happened

We are currently updating our API using a completely new backend. As part of this work, some bugs have appeared in the migration process. Due to the above, no new resources (repositories, environments, and files) could be created in Integrates between April 6th and 7th.

What we’ve done

When we detect the errors causing the problems, we fixed them in successive Integrates versions.

What’s the impact

• Between 2020-04-06 at 11:28 am and 2020-04-07 at 05:01 pm (13.5 working hours), 18 failed creation attempts were made in Integrates.
• This issue affected 0.8% of our users.

What we are doing to help

We are improving our test cases to prevent further problems like this one.

What happened

We are currently updating our API using a completely new backend. As part of this work, some bugs have appeared in the migration process. Given the above, some findings could not be accepted at Integrates between April 3rd and 6th.

What we’ve done

After we detect the error, we fixed it by correctly validating the user inputs.

What’s the impact

• 6 failed acceptation attempts were made between 2020-04-03 at 03:42 pm and 2020-04-06 at 08:14 am (1.5 working hours)
• This issue affected 0.2% of our users

What we are doing to help

We are improving our test cases to prevent further problems like this one.

What happened

On 2020, from March 26th at 2:00PM to April 2 at 6:47PM (a total of 52 office hours), Integrates suffered of performance issues due to bad table design for the authorization model, enabled after three commits: b44cfa0, 79c48ba, and 6419323:

What we’ve done

We designed a more lightweight database adapter 2, and introduced a cached authorization model in b14adb0 1 and e5d8445 1. Finally, we introduced the best possible table design in a826210, recovering the expected performance levels:

What’s the impact

During 1 week, our users experienced slowness while navigating the application.

What we are doing to prevent this

We added 1200 mock users to the development database in order to early test the application performance.
Also, there is in course a big migration that puts our back-end to work in an asynchronous fashion which highly boosts those operations that have to do with simultaneous database access and retrieval.
We are also always monitoring production systems through a suite of different services, so we can redirect the development towards more performant solutions.

What happened

On April first, 2020, after some weeks migrating the access control system on Integrates from a single-tenant architecture to a multi-tenant one, we received a report with a small list of users that suddenly lost access to the platform.

What we’ve done

We believe the problem was originated due to a slightly deviated migration script. So we ran an integrity check over the database to verify that every single user had the required fields and access attributes set and found a few with some of them missing.
Once we had the list, we manually filled those attributes, reestablishing their access.

What’s the impact

To our knowledge, 0.028% of the users were affected:

• 0.009% were reported by the affected users.
• 0.019% were fixed proactively (before users noticed it).