In this blog I’ll be sharing my understanding of Lean and Agile Principles in Software development as well as the need for Scaled Agile Framework.

As we know, software development is about developing products and IT applications. The most common process traditionally followed is the waterfall model of Software Development Lifecycle (SDLC) to deliver the project. Project delivery is about performing sets of operations for various stake holders. Historically, the typical target project duration has been around 9 months. In my experience, I have not seen a single project executed successfully within this timframe. Invariably, there has to be change in the scope, cost, or quality in order to ensure the delivery schedule doesn’t change. The most common reasons I have seen for these changes, which actually represent failures, are

  1. Scope changes
  2. Poor requirements understanding
  3. Incorrect estimations
  4. Tangled team dependencies
  5. Change in business process by the time project delivered.

The standard set of project operations during project delivery are significant contributors towards project delivery lead time. The focus for lean principles is to eliminate the waste by removing some operational overheads and increasing process efficiency. In order to identify forms of waste one needs to come up with a value stream (eg: linear equation in mathematical modeling). The biggest problem for software development organizations is to create this value stream of digital and intangible outcomes.

I have a theory on the history of Agile: Agile has evolved as framework process principle by applying Lean on the standard ( inefficient ) waterfall development process.

Lean development can be summarized by seven principles which very close in concept to lean manufacturing principles:

  1. Eliminate Waste
  2. Amplify Learning
  3. Decide as late as possible
  4. Deliver as fast as possible.
  5. Empower the team.
  6. Build integrity.
  7. See the whole.

All these principles are directly and indirectly implemented in Agile. As such, Agile principles basically “follow the waterfall process but in shorter cycles”.

There are multiple frameworks that are complaint to Agile principles, the most popular of which is the Scrum framework. Based on scrum, sprints are time bounded ( recommendation is 2-4 weeks). Product Owners, Scrum masters, and scrum teams work closely together to complete the sprint.

The Product Owner is the person who is responsible for consolidating requirements into a Product Backlog. The Scrum Master looks into the product backlog and decides on priorities, which are codified into a collection of stories that team will commit to implement (burn down) within the sprint duration. Teams have daily standups that cover what they did the day before and what are they planning to perform that day to ensure that the sprint is on track. At the end of the sprint, during retrospection meetings, teams check what improvements can be added in future sprints. As such, the release plan is similar to traditional project plan but without extensive efforts. It is a very high level plan with tentative dates. A release contains multiple sprints, and based on the progress of the sprints the release plan shall be updated. The focus should be more on sprint planning than release planning. The release plan is more to provide high level visibility to management and product owners.

Now that I have introduced Lean Principles and Agile terminology, some interesting artifacts of the evolution contained in my hypothesis become clear :

  1. Eliminate Waste : Product manager works with team in breaking the stories and coming up with business value. He works with the engineering team to get the time cost for completing the story. Priority is decided mostly on these two values. Scrum team takes up high priority stories and effectively eliminates lots of waste.
  2. Amplify Learning : The agile retrospection meeting is a valuable resource to carry over learning to the next sprint plan.
  3. Decide as late as possible : Sprints with 24 weeks will give the flexibility to make decisions later.
  4. Deliver as fast as possible : Once the story is picked moved from the product backlog to the sprint backlog, turn around to delivery is short.
  5. Empower team: Since the scrum team is selecting which stories will be pulled from the product backlog to the sprint backlog using story priorities, management is not making decisions on what should be part of the sprint backlog.
  6. Build integrity: Agile core values ensure team members integrity.
  7. See the whole: Release plan makes the scrum team approach projects holistically while still maintaining focus on the sprint.

Success of the Agile implementation depends a lot on organization mindset, especially leadership maturity. Using tools is also necessary ( eg: JIRA Agile + Confluence ) which helps in recording the all artifacts. Continuous Integration, Continuous Delivery (eg: Stash + Bamboo) and Test Automation complement the value Agile creates. Effectively Agile Scrum framework would reduce the failures I have highlighted above to the maximum extent. However, there are still problems with an Agile Scrum framework:

  1. Focused on Engineering team : Though the product owner is from a Product Marketing team there is no established framework on how the product backlog should be groomed. Usually there are multiple product owners with requests that need to be included in the product backlog. For product log grooming, I have recommended Kanban process to many customers and they are very happy with the solution. This way the input from Service Engineering to product backlog is still an open area.
  2. Matrix Larger Engineering project teams adaptability : Agile recommends 6-8 member to a team. In enterprises, the projects I have worked on usually number around 20 – 50 members at most global and matrix organizations. Some principles of scrum work to some extent within Engineering, but when representing an engineering team in cross functional project meetings (Marketing, Engineering, Systems, Service Engineering and Operations, Sourcing, QA, RA, Program Project Manager) agile and scrum process aren’t understood. In these situations adaptability is a key factor. You have to prepare yourself differently for different meetings.
  3. Bigger Platform change : Sometimes OS announce End Of Line or Strategic decisions that drive OS changes and product platform migrations. To the customer it doesn’t make any difference since they care about application functionality. As per Agile, we need to focus on changes that makes difference to customer. Can these projects be neglected?
  4. Design focus : There are side effects of doing design in the sprint, especially for the products which need to be scalable in nature. Platform design is very critical in the longer run and in the rush of delivering in shorter cycles this should not be compromised as it will prove very costly in the long term.
  5. Products incremental release to multiple instances : Continuous Delivery works really well for a single instance eg: Facebook (push model). Releasing to external teams after every sprint makes sense, but imagine updating a high volume install base. People argue that companies like Microsoft release patches service packs very frequently; It is their process. However, an Agile framework doesn’t cover this. Probably they left this as it is part of continuous delivery scope. Upgrading systems which are under regulatory surveillance is different ball game.

Software development Enterprise needs are probably addressed by Scaled Agile Framework (http:scaledagileframework.com) and deeper looker is needed. In the next blog entry, I will elaborate on this. As lean emphasizes making improvements on existing processes to satisfy different challenges and refinements to existing or new frame works, it will keep evolving, and I’ll keep writing.