What happened

• We are currently improving our security by enforcing more strict validations. As part of this work, repository protocol input validations were modified to prevent unexpected queries to our API.
• After this change was introduced, we ran a script to assign a protocol to the old repositories that didn’t have one. In the process, we accidentally removed all the repositories from a certain number of projects.

What we’ve done

• We restored the project's database table using a backup from 2020-05-11 at 10:50, one made shortly before we ran the script. With these, we managed to revert all the changes and restore all the deleted repositories. The script restoring all the repositories was run on 2020-05-12 at 15:37.

What’s the impact

• Yesterday, May 11 at 11:10, we ran a script to update some registers in the database. After that, we noticed that a total number of 1050 repositories from 27 projects had been deleted. The time elapsed between the mistake and its fix was 13.5 hours.

What we are doing to help

• From now on, all the scripts, all the scripts that affect production data will be versioned in our repository, through a merge request process.
• We’ve had a very strong backup policy, so we can easily revert our data to a previous state.
• This issue did not affect the security tests nor the database that we use in our testing process. We are currently improving our security by enforcing more strict validations.