What happened

• Last Tuesday (January 7th) at 12:03 PM there was an incident in which two of our customers using the Breaks service got their pipelines broken due to findings with ‘Accepted’ treatment.
• The commit with the changes that affected the pipelines was merged at 09:43 AM and its details can be found at https://gitlab.com/fluidattacks/integrates/commit/7913964
• We implemented a new functionality which restructures finding attributes so that now information concerning treatment data is stored inside a historical register. This feature allows to store and keep constancy of all treatment changes with its concrete data.
• This is a very sensible change given the number of parts of Integrates that we had to modify in order to make this new feature effective. Specifically, the change modified the API signature. The Breaks service uses the API and the signature was not updated, breaking the expected behavior.

What we’ve done

• January 7th at 4:41 PM, the issue was fixed. The affected customers confirmed that the issue was solved at 5:57PM of the same day.
• The way we solved the problem was updating Breaks service, due to the finding treatment change mentioned above, to retrieve the treatment correctly.

What’s the impact

• Two of our customers got their pipelines broken because of those findings whose treatment were ‘Accepted’. The problem lasted 4 hours and 38 minutes.

What we are doing to help

• In order to avoid future similar issues we are going to implement ‘versioning’ for our API. This way, future incoming changes affecting the API will automatically activate the corresponding changes in breaks.
• With this announcement, customers are being notified that this was an internal error due to a set of changes we made in the data structure and what we did to fix it.