I work in Information Technology at a large company. We, like many companies around the world, are aggressively adopting Agile techniques to build and support software applications within the company. The basic principles of Agile are outlined in The Manifesto for Agile Software Development, developed in 2001 by a group of software developers who had been practicing the techniques for many years. Agile and Agile Transformation is one of the hottest management transformation fads right now (think back to Business Process Re-engineering or Strategic Sourcing which were the big ones in the 1990s).

This being a fad, the term “Agile” gets used to mean many things. There are many methodologies with their owners and proponents. These include Scrum, Extreme Programming, Kanban, SAFE, LeSS, etc. I will try to explain what I understand Agile to be in principle and stay away from the specifics of the various methodologies.

Agile is based on the principle that you get better results and value from small dedicated teams working on small increments of work that are quickly deployed to production for use by the customer. The team works with customers and defines a backlog of features for the application or product. The key stakeholder in the product, often called the Product Owner, sets the priority of each item in the backlog. The team then decides what features will be taken from the backlog for the next iteration of product development. The team manages itself and has all the skills and tools to deliver working code in production. Ideally, line management is spread thin and focused on eliminating blockers outside the team and obtaining the resources the team needs.

In one version of larger-scale Agile, LeSS or Large-Scale Scrum, they advocate no budgets, no projects, no programs, no project or program managers, even no managers. The team and the product owner maintain a longer-term roadmap, but that is continually revised at each iteration as feedback from new features being introduced changes priorities – new things are learned so the roadmap changes. Change and adaptation are constants.

Doesn’t sound very orderly, does it?

So why do most software companies and now many IT groups use Agile for development? Let’s talk about what it replaces. Software projects typically followed a waterfall method. You’d start off with high-level requirements which would be the basis for a huge Microsoft Project plan and a detailed budget. Then the team would assemble, with analysts working with customers defining detailed requirements. Once the requirements were complete and signed off, the analysts would write specifications. Programmers would code from the specifications and then testers would test. Finally, after months or years, the end result would be delivered.

This end result was almost always over budget and schedule and did not meet customer expectations. To quote the consulting firm McKinsey from 2012 “On average, large IT projects run 45 percent over budget and 7 percent over time, while delivering 56 percent less value than predicted.” Because these projects are large and complicated, we created the illusion of control through layers of management and a big project plan that was pretty much a work of fiction.

It may seem orderly, but it certainly didn’t produce results in an orderly manner.

The seemingly less-orderly Agile approaches, when properly implemented, deliver far better results. By continuously incorporating feedback, the right things get built. By empowering individuals and teams, things get built right. Much more orderly in the end.

Changing the way an organization works is not easy nor should it be taken lightly. Perhaps the biggest underlying change is one from Theory X management to Theory Y management (from The Human Side of Enterprise, Douglas McGregor 1960). Theory X says people hate work and will try to avoid it and so need to be coerced, controlled, and directed so they will work. People will not take responsibility and want to be directed. Many management practices – performance reviews, individual targets, bonus systems – are based on these Theory X assumptions.

On the contrary, Theory Y assumes that people will work as naturally as they play or rest. They will be self-directed for goals to which they are committed. In the right environment, people seek responsibility and will creatively apply themselves. Agile practices are built on Theory Y assumptions.

Changes the practices and attitudes of people across the organization is difficult. Managers need to give up some control. Executives need to accept a little more uncertainty. People doing the day-to-day work need to believe in and adopt the core Agile practices.

I was a skeptic several years back. Now I am a convert and deeply involved in our transformation. It is a journey, not an event. We are not perfect but are progressing. We are pursuing it in an orderly manner, inspecting and adapting as we go.

(Credit to Large Scale Scrum: More with LeSS, Craig Larman and Bas Vode, 2016 for the management theory discussion)