What happened 

• In normal conditions, an email is sent to the users to let them know about a comment made by our team with the result of a requested re-attack. This applies for both Machine and Squad plans. Currently, we are working on giving more useful information in the comments for re-attacks performed by Machine. Due to an error in the logic, the comment was being made for every vulnerability found by Machine, whether it had a re-attack requested or not.

What we’ve done 

• We reverted the change that brought this flaw in the code. We also stopped all Machine executions that were running with a version of the code which had the bug.

What’s the impact 

• Over 15.000 emails were sent to our customers whose code was analyzed by Machine before we detected the issue.
• Some findings may have a lot of comments made by Machine.

What we are doing to help 

• We are going to review the bugged logic to fix it, and we are going to increase the test cases before deploying it to production.
• We are going to track the added comments and delete them from the database, so the users Consulting section is not full of useless comments.

What happened

• Due to a testing strategy implementation to reproduce production tests in local environments, a conflict in service ports unleash in a service outage.

What we’ve done

• Revert commit that were causing the conflict.

What’s the impact

• ARM was unavailable  from 2021-10-22 16:11PM until 2021-10-22 16:30 (19 min).

What we are doing to help

• Pin production to a specific commit, not to master branch. Preventing rollout issues.

What happened

• Due to a recent DB migration, some vulnerabilities are not loaded correctly, caused by an inconsistency in migrated data.

What we’ve done

• Improve ARM logic to prevent errors due to those inconsistencies.
• Update inconsistent data from backups.

What’s the impact

• Accessing 40 groups in the ARM was intermittent from 2021-10-05 12M until 2021-10-05 16:40 (4.5 hours)

What we are doing to help

• We are currently ensuring any further changes to our database have a full backup.
• Continuous monitoring to data consistency.

What happened

• Due to a recent feature implementation in our authentication engine, we refactored some variables that conflicted with the ARM cache, preventing access to some Groups for a while.

What we’ve done

• We immediately invalidated the faulty cache and fully restored access to the ARM

What’s the impact

• Accessing the ARM was not possible for a few minutes. Between 12:05 and 12:24.

What we are doing to help

• We are currently ensuring any further changes to our authentication engine are done in a safer and more spread out manner

What happened

• Due to a recent feature implementation in our AI engine, we launched a new job that made a very huge set of parallel calls to our API, degrading its performance for a while.

What we’ve done

• We immediately stopped the involved job and committed a first fix (https://gitlab.com/fluidattacks/product/-/commit/395ca6b2070382fcc76f764922520e5999836b65).

What’s the impact

• Accessing the ARM was not possible for a few minutes.

What we are doing to help

• We are currently ensuring we don't overload our API with lots of intermittent expensive calls and also optimizing how our AI engine communicates with the ASM.

What happened

• Due to a session management change (https://gitlab.com/fluidattacks/product/-/commit/1f67cd266283a8846ab8fc84ed761a3b89d2ff33) user cookies were deleted for current session.

• If a user sign-in with an existent session all cookies are deleted and an unauthorized error is propagated through all user sessions. The bug was injected on 2021/01/27 14:31 (EST).

What we’ve done

• Now the cookies are not deleted, only a notification about concurrent sessions. (https://gitlab.com/fluidattacks/product/-/commit/b8600cc80d0b4fd5142a02c7f0d71b564f68bb25)

What the impact was

• Users who tried to log in with existent sessions get unauthorized error.

What we are doing to help

• Check all user sessions to confirm if any user is affected. 

What happened

• Due to a large recent cache migration, we didn't figure out how many new Redis connections would appear, so our connection limit parameter was not updated.
• That caused some Redis connections not to reach the endpoint between 2020/12/17 13:30 (EST) and 2020/12/17 15:20 (EST).

What we’ve done

• We found the right value for this parameter and committed the fix (https://gitlab.com/fluidattacks/product/-/commit/03730d0c95ddd84cde5771ed91560504518866fa).

What the impact was

• Vulnerability loading for some big projects was not working.

What we are doing to help

• We currently ensure that we properly adjust each existing parameter of our Redis instance for every incoming cache change.

What happened

• We are currently working on migrating our backend through Starlette.
• On 2020/12/03 at 17:15 (EST), our team removed old configurations from Django and Kubernetes cluster. We didn't notice that the main redirection / was associated with Django.

What we’ve done

• After down notification, our team implemented a temporary fix moving / redirection to the Kubernetes ingress. We recovered service on 2020/12/03 at 18:05 (EST).
• We approved the final solution on 2020/12/03 at 18:19 (EST).

What the impact was

• Users were unable to log in from 2020/12/03 17:46 until 2020/12/03 18:05.

What we are doing to help

• We continue standardizing our backend to Starlette.
• We continue debugging the process and making tests for all functionalities in search of unexpected path problems.

What happened

• We are currently working on migrating our backend through Starlette.
• On 2020/11/20 at 15:41 (EST), our team moved the pkg to the new back. We expected this change would not affect anything, but on November 23 at 11:11 (EST), we noticed that the Executive reports weren’t generated because some paths didn’t match with the new structure.

What we’ve done

• After a debug process, our team reproduced the issue at 11:20 (COT) on 2020/11/23.
• We approved the solution on 2020/11/23 at 13:26 (COT).

What the impact was

• Users were unable to generate reports to Integrates from 2020/11/20 until 2020/11/23 at 11:30.

What we are doing to help

• We continue to standardize our backend to Starlette.
• We continue to debug the process and make tests for all functionalities in search of unexpected path problems.

What happened

• We are currently working on migrating our backend through Starlette.
• We noticed that the emails sent between 2020/11/20 and 2020/11/23 redirected to an unauthorized view and closed the session. With this transition to Starlette, we use a new path with /new, which causes some functions to be no longer available in the old path.

What we’ve done

• After a debug process, our team figured out the issue on 2020/11/20 at 18:10 (COT) and worked on the fix.
• We deployed the solution on 2020/11/23 at 09:32 (EST).

What the impact was

• Users could not use the links sent on emails that redirect to Integrates from 2020/11/20 until 2020/11/23 at 11:30 (EST).

What we are doing to help

• We continue standardizing our backend to Starlette.
• We continue debugging the process and making tests for all functionalities in search of unexpected path problems.