Last week, a backup incident in a staging server on GitLab resulted in the deletion of the production database and was responsible for 6 hours of data loss and some server downtime.
The official blog post about the incident starts like this:
Yesterday we had a serious incident with one of our databases. We lost six hours of database data (issues, merge requests, users, comments, snippets, etc.) for GitLab.com. Git/wiki repositories and self-hosted installations were not affected. Losing production data is unacceptable and in a few days we’ll publish a post on why this happened and a list of measures we will implement to prevent it happening again.
As we all know, data loss is a major nightmare for any product out there, but it’s worst if you’re a cloud code repository with a massive amount of daily users like GitLab. Although these things were not supposed to happen in 2017, organizations are made of humans (at least for now…) and making mistakes is part of being a human. But is the way that the organization deal with the problem that makes the difference.
GitLab’s Approach After Data Loss Incident
GitLab’s approach was based on transparency and it’s getting some positive feedback from the community.
They didn’t try to hide the problem and instead, they set up a live stream of the team resolving the problem (8 hours) and released a google docs explaining step by step how the mistake happened and how it got solved.
This technical calculator consists on a series of questions that will help you identify the current technical state of your startup. And possibly identify a few red flags that could become a bigger problem in the future.
Manoj Bhargava is an entrepreneur, philanthropist, and founder and CEO of 5-hour Energy company. His net worth is over $4 billion, but the most interesting part about him, is that he is giving away more then 90% of it to build a better future for the world thought the movement Billions in Change.
Billions in Change is a movement to save the world by creating solutions, through technology and innovation, to the most basic global problems – water, energy and health.
They are working on several projects right now like:
Free Electric – A hybrid bicycle that you pedal for one hour and have electricity for one day.
Limitless Energy – Using graphene to conduct energy, in the form of heat, from underneath the earth, pollution free.
Rain Maker – Produce fresh water from sea water.
Renew – An ECP (External Counter Pulsation) device that can make people healthier by assisting the hearth pulsations and reducing heart problems
But I don’t want to spoil too much of the film because it really worths being watched!