What happened

• We are currently updating our API 2 using a completely new backend. As part of this work, and seeking for creating a faster API, a commit 2 was introduced with the goal of improving some technical aspects of data fetching and concurrency handling. Although the change accomplished the desired effect, the current in-memory cache data was not updated to reflect the changes performed.
• This led to unavailability of parts of the application retrieving information using the method introduced by the change.

What we’ve done

• After identifying the problem, the cache was purged to allow filling it with the new data structure

What’s the impact

• The change was deployed to production at 10:28 am of April 23. 12 minutes was the time between the change was deployed to production and the cache was purged.
• During that time, 13 users were affected by the unavailability of retrieving parts of the projects information.

What we are doing to help

• Those kind of changes are rare. However, we identified the pattern of code changes that may affect the in-memory cache structure and defined means to purge the cache when this happens.