Mastering CI/CD: From Concept to Implementation

Which environment is adding value to the company? That's correct, production. Why do we invest the majority of our development time on environments other than production?

How did we end up in this situation? Probably due to a series of short-term victories that seemed reasonable at the time. Every business begins on a small scale. We encounter an issue in production. Instead of repeating the mistake, we opt to develop a test. Later, as we grow, we break a few more things and add more tests, but complexity multiplies and we are always playing catch up. We create non-production environments to catch these issues before they hit production. Despite this, production still breaks, and, as our team expands, so does the complexity of our products and our environments. We begin spending a large chunk of our time fixing inconsistencies between environments and bugs in different environments. Most failures that we've seen in the past are covered with ever growing suite of tests, but our bugs keep propping up due to new discoveries. We keep at it, despite the fact that none of this work creates value since large majority of changes aren't happening in production!

So we build CI/CD...

There are numerous tools available in the market that enable us to automate our development process from commit to production, ensuring that our modifications undergo testing in all environments before being deployed. This initial step is challenging but crucial, as it initiates the flow of beneficial changes directly visible to customers. Our extensive CICD system, coupled with the intricacies of our product and testing procedures, sometimes results in delays of hours before a release can be made after passing through all stages. In case of any blockages, we must activate the "Andon cord" to halt the delivery of value and fixes to production and address issues in the complex pipeline.

Why not take it to its logical conclusion?

Each company has the potential to improve production processes directly, enhancing safety and reducing errors. The choice of technology stack and methodologies should align with the engineering culture of your organization. Envision transitioning from concept to value within minutes. Picture responding to customer feedback promptly. These aspects are what truly enable the value of a production-first engineering approach.