DevOps is an emerging trend in the IT industry, so it does not yet have a definitive standard, framework, or body of knowledge that is universally accepted. Consequently, different stakeholders view and interpret DevOps differently.
I believe DevOps is a cultural shift that emphasizes collaboration, communication, and integration across the enterprise in order to break the impact of organizational silos, with the objective of eliminating all operational waste, keeping all the stakeholders on the same page, and enabling continuous delivery. Continuous delivery allows us to deliver updates to systems as often as necessary while avoiding mistakes and incidents.
Operations waste has always bothered me in my various roles as a developer, tech lead, project lead, and project manager. Making mistakes that result in incidents, defects, and rework adversely impacts both quality and productivity. As a result, I have come to appreciate the role DevOps plays in tackling this problem. In addition, I believe that focus, visibility, trace-ability, and compliance are the other key elements that play a vital role toward adopting DevOps. Organizations need to have the right set of tools and technology, along with automation, to overcome these problems.
In my twelve years in this industry, I have faced these and similar challenges many times. This article is about how I learned to address them using some lessons from my journey to understand and drive DevOps adoption.
Started my career as a trainee in 2003 with a large engineering enterprise but in a small IT team of only six members. I was one of the developers responsible for delivering web applications for the business. The development involved using what was then the latest technology, the J2EE stack. We used to spend 80 to 90 percent of our time designing and coding. It was exhilarating to do something hands-on, and I began to enjoy my work a great deal.
I moved to a large health care company in 2005, where there was a strong emphasis on software processes. This brought me in touch with different facets of software development that I had previously been unaware of. I still recall one of my senior managers cautioning me about the development overhead in a heavily process-oriented company. The on boarding effort in this group took more than three months and covered a mind-boggling number of topics, such as the health care domain and FDA rules, the Digital Imaging and Communications in Medicine (DICOM) standard, design history, Six Sigma process improvement, and version control and defect tracking.
I felt lost initially, and there were many challenges. For one, there was a lot of operations waste like it took nearly two to three hours of unit testing for a code change that barely took fifteen minutes! There used to be several iterations for a production release, each spanning two to three months, for systems integration and testing. We often ran into merge issues and build and packaging failures. Integration testing was a long and slow process. In order to incorporate the feedback it used to many late nights.
During my first year at this firm, I don’t think I did coding even 25 percent of the time. But the good part was I became familiar with a system perspective, which is the DevOps approach of looking at the entire system.
As a tech lead, I was given the responsibility to lead the delivery of two features. For the first feature I worked within the development team, finalizing the requirements and then kicking off the development effort. My team lead drafted an MS Word template that we used to capture the requirements. What followed was a nightmare. Collaborating and consolidating comments from each individual stakeholder across different functions (systems, service, marketing, quality, legal, etc.) and making sure everyone agreed on what was captured was extremely difficult. We had to open multiple review docs and consolidate all the comments provided by them. It was an absolute waste of time. Review cycles were longer because no one understood each other’s perspectives, and throughout this effort I was responsible for updating and circulating the updated version. It took almost three months to get approval from all stakeholders for the first feature to move to the design phase. By the way, it was the first feature in the program to get approved. Meanwhile, the second feature got dropped.
In 2007, I moved within the software applications division to become a lead software integrator and started my progression from the development team to IT operations. I was responsible for keeping the sub-functions / teams on the same page. This was their best in class for process adherence. Although we did not have an official agile label, we had weekly sprints and other iterative practices. In rough numbers, we had twelve million lines of code, four hundred thousand files, seven thousand derived objects, and close to ten different programming languages. The build and release cycle was completely automated, which used to take more than twenty hours. We had highly scalable and version-controlled usage and architecture.
We also had a fresh release for every program and used to be available every week. I had driven the integration of second-party platform components from other teams into all parallel programs. At the same time, we had nearly ten major programs that required many manual steps for the integration. We automated everything possible, which made 60 percent to 70 percent of my time available to focus on other organizational priorities.
Soon I realized that I was in more of an operations role, with ~ 60 percent of my work involving MS Office tools and, of course, emails. The rest of my time, I was coordinating with individuals or teams to get or provide visibility. It was frustrating to see how unproductive the MS Office’s so-called “productivity” tools were. I longed for an evolved version of these tools that would make my life simpler rather than more difficult.
We attempted to build a system for real-time visibility to eliminate operational overheads. I was assigned a trainee for executing a proof of concept before allotting more resources. We started what was essentially a marathon, with the idea to come up with a system that integrates different activities; documents reviews, development, build and drops, infrastructure, test plan, and execution.
I did not realize that I was trying to solve a very big problem; I later learned it’s called the integrated Application Life cycle Management! The sheer complexity of the task at hand meant a proof of concept that met the business requirements never materialized, so it was abandoned—along with my nearly year long investment in it. I guess even today, companies struggle to get the integrated ALM streamlined.
Compliance taught that there are times when I have no choice other than to adhere to the rules and regulations that are required.For example, we had to build the complete trace-ability of a design history file on an Excel spreadsheet for a big project as part of our FDA audit preparation. Just gathering information from different stakeholders was a huge task, not to mention discovering gaps and the work required to fill those gaps. Here again, I believe using the right ALM probably would have made it a more automatic, streamlined, and less tedious process.
In the health care industry, systems verification is one of the most time-consuming activities. Team members had to manually sign every single test case executed with a build label, system information, and the date. There used to be a different procedure to verify that the system was production-equivalent. We interviewed more than fifty engineering managers to check whether there was a better way to overcome this problem. The business case required changes to adhere to the IT controls mandated for regulatory compliance defined by the QA team. Now I see there were a lot of other options to overcome this challenge. The effort, although significant, would have been worthwhile, given that it formed a minuscule part of the whole product life cycle span.
All these experiences were very tough at the time, but now when I look back, I realize they helped me understand the view of both developers and IT operations—and, ultimately, in designing DevOps transformations. My current focus is achieving visibility, trace-ability, and compliance along with the DevOps elements of integration, collaboration, and communication, with an agenda of eliminating waste for customers.
In my current position I have found great value in our use of Atlassian tool set. We use their tools internally and provide services to customer. Good tools help you improve collaboration and communication and deliver the best results. Organizations that want to adopt DevOps need to choose their tools wisely, because one size does not fit all and there are no ready made solutions. Smart organizations will not hesitate to invest in building the best automated integrated ALM solution in addition to hiring professionals who have both development and operations experience.
I’m sure many of you have had similar experiences. Do you think I missed out any other critical elements? Please share your thoughts in the comments section below.
This blog is originally posted at cmcrossroads