“If you haven’t broken the production, you’re not really a developer”

by time news

“If you haven’t broken the master, you haven’t lived. Or you’re not really a developer,” says Aviva, director of the Wix backend guild. Many of us can relate to this statement. We all break the master, we all upload bugs to production.

But it’s no coincidence that we all broke the master, and that we all had fun with it too. Thanks to the CI/CD revolution – instead of releasing a version once a month, code we wrote in the morning can meet our customers even before lunch (thanks for this opportunity to DevOps!)

“When we started this company it was clear that from the first day there was full CI/CD” says Tal, CTO at Torii. “I mean, before we even wrote the POC of the first thing, now to have something that, however you build it, launches it on production, so we started from a good base, but there are a lot of things to do in the end, once the CI/CD is real, That the developer finally comes and presses a button, what’s important to me, is both for myself and for the developers here, it’s not to sweat that I press the button, to know that as soon as you press it and it reaches production, reaches the customers, things are going to work.”

Itay adds on the importance of a fully automated CI/CD process, “If you have a fear of deploying at any point, it means there is something amiss. It means you need to understand where the fear comes from and address it.”

And Miri, development manager at AppsFlyer, emphasizes that the CI/CD process should include both parts, both CI – continuous integration, and CD – the process in which the products are taken and uploaded to production: “Mainly the part that I think is missing in most places, is the CD part The Continuous Deployment part, in my eyes, is the part that makes all the difference. The Deployment is not a hassle, the Deployment is really part of the CI/CD, it’s the same pipeline, it runs together, it finishes together. If the CI will succeed, then the deployment also happens and it doesn’t require more tools and more fiddling from me.”



The last point is very important. Because, alongside the great advantage, rapid release of versions increases the chance that bugs will find their way to production, and we must make sure that the deployment process includes protections that will reduce the risk, such as feature flags and gradual roll-outs, as Assaf from Wix explains: ” You say, I changed something in my product, and I want to expose it only to a certain percentage of people. If this percentage has received the relevant configuration that will expose the change, I slowly expand this percentage, and I roll out the change. In the end, I actually lower the fence “.

“Full of gatekeepers” sum up the point Gali and Karen from Taboola: “We have a process in the team that checks if we have broken the master, which compares one machine that has a new feature against a machine that has Brillis.” Another example of such a gatekeeper is the Canary process: a gradual version release process designed to minimize the damage in the event of a malfunction: “The idea is instead of releasing to production 100 percent, we release to 10 percent, 20 percent, or really to specific customers, and test it first Just to see that things work as they should, that the new things we wrote do have the impact we expected, and if so, and if not, then we actually either release 100 percent, or do a maximum rollback and really fix and make a new one.”


Hey, there are more great episodes of Dark{Mod}


But despite all this, bugs were and always will be. And if there is already a bug, just make sure it’s easy for you to roll back the version.

“Eight years, even more, we don’t do hot-fixes, there’s no such thing,” emphasizes Itay from Wix. This is the meaning of a functioning CI/CD in his opinion: “There is no such thing, what is a hot-fix? First of all, first of all roll-back. This is the first law in production: first of all roll-back”.

And what about releasing second versions before the weekend? In the episode you will hear about many stories that will make you consider it again. We’ll just conclude and say that if you don’t have a complete CI/CD, you don’t want to be the one waking up your friends on the team on Friday night.

In conclusion: it is impossible to exaggerate how much CI/CD affects our development experience. So if you are not satisfied enough with the version release process at your company, it is time to knock on the door of the DevOps and ask them to upgrade your development experience.


Dark {MOD} is a docu-comic network series for developers. Every two weeks we will upload a new episode on the YouTube page And in an extended version here at Giktimes.

You may also like

Leave a Comment