Untangling Your Organizational Hairball: How to Increase Local Autonomy

Communication is a sign of dysfunction. It means people aren’t working together in a close, organic way. We should be trying to figure out a way for teams to communicate less with each other, not more.

—Jeff Bezos, Executive Chairman of Amazon

Traditional organizations focus on optimizing efficiency, trying to specialize and maximize the use of each resource. In contrast, modern agile organizations prioritize effectiveness and speed first and optimize efficiency second. In optimizing speed, particular attention is paid to reducing dependencies between organizational units. Some go so far as to classify dependencies in organization, communication, and technology as defects.

This shift marks a fundamental change in how large organizations approach their organizational setups and requires a reassessment of team structures, collaboration models, and system architectures.

Understanding Dependencies

In large organizations’ complex ecosystems of software development, two types of dependencies are common: technical and organizational. Technical dependencies are most common in legacy systems, where monolithic architectures form a tight-knit network of interconnected modules. Even the smallest change can require a cascade of adjustments throughout the system, leading to longer development cycles and an increased risk of error.

Organizational dependencies usually result from the best intentions of an organization’s structure and culture. Departmental silos, coupled with rigid processes, can create bottlenecks in decision-making and inhibit innovation. These dependencies often manifest in delayed approvals, cross-departmental coordination issues, and lack of alignment.

Technical and organizational dependencies can be related. Technical dependencies can exacerbate organizational challenges as teams become bogged down in the complexity of the legacy systems they manage. Conversely, a rigid organization can hinder the adoption of more flexible and modular technical architectures, such as self-contained systems, which include their own UI, logic, data, and even infrastructure. Addressing these dependencies is critical for organizations that want to improve their software development agility to respond more quickly and effectively to market demands and technological advances.

Addressing Technical Dependencies: From Monoliths to Microservices

Confronting technical dependencies in large enterprises often begins with a strategic shift from monolithic architectures to more modular approaches, like microservices and micromonoliths. Monoliths, characterized by their large, singular codebases, often lead to complex interdependencies, where changes in one area can ripple unpredictably across the entire system. This tightly coupled structure not only hinders rapid development but also escalates the risk of system-wide failures from minor code alterations.

Moving to microservices architectures reduces technical dependencies by segmenting applications into distinct, manageable units, each with its own functionality and specific APIs. This transition results in dependencies between microservices, but these are more clearly defined and observable. This clarity of dependencies fosters a loosely coupled architecture where teams can develop, test, and deploy services independently. This methodology greatly streamlines agility and minimizes coordination complexity, transforming potential dependency challenges into a structured, manageable architecture that increases overall system resilience and flexibility.

An intermediate step toward this architectural paradigm is the adoption of micromonoliths. While retaining some characteristics of a monolithic architecture, this approach begins the decomposition process by modularizing certain aspects of the application. It is a practical first step for organizations to adopt microservices, gradually acclimating to new structures and processes.

These modular architectures not only facilitate a more resilient and scalable system but also empower development teams with the autonomy to innovate and adapt swiftly, which is crucial in today’s rapidly evolving technological landscape.

Tackling Organizational Dependencies and Streamlining the Software Value Stream

Addressing organizational dependencies in large enterprises requires a deliberate restructuring of teams and processes to foster greater autonomy and agility. It is essential to look at the software value chain—holistically, from beginning to end.

I sometimes do this when talking to AWS customers about their software development processes. I ask them to show me a scrum or Kanban board. I’m particularly interested in the board’s last column. 99% of the time, it’s labeled “Done.” The conversation usually goes like this:

I ask, “What does ‘Done’ mean? Does every user utilize the software productively?”

“No. We just finished development. The software still needs to be integrated.”

“Will it be used after the integration?”

“No. Then it has to be tested and released—provided all the bugs, or at least the priority ones, have been fixed.”

“Will all users be able to use it then?”

“No. It still has to be deployed, load tested, accepted, and released by our service provider in the production environment. But then the users can use the software.”

“How often is each process step performed?”

“Integration is done once a month—testing and release as well. But it often takes longer. We release to production once a quarter, except in Q4 due to Christmas. So three times a year.”

It’s unfortunately similar when I ask what happens upstream before software development starts. The first column on the board is usually labeled “To Do” or “Backlog.” You might think this is where the team collects, evaluates, prioritizes, and works on all ideas. That is not usually the case in large organizations, even with agile software development.

Engineers first clarify the requirements for new ideas so an innovation or steering board can prioritize them. Business analysts then write a business plan for which a budget must be requested, reviewed, negotiated, reduced, and approved. A solution architect creates the IT design based on the requirements and business plan; then the development team implements it. Prioritization is done once a quarter. Business cases are written continuously, as resources are available, and approved quarterly. Budgets are created annually. IT roadmap decisions are made monthly—all in a “thoroughly agile organization.”

From a software development perspective, the organization might look agile and fast, given the team releases software every two weeks. But from the end customer’s point of view, it could take a year to implement an urgent request.

Streamlining the software life cycle in large enterprises extends beyond development and deployment. It encompasses a holistic approach to refining all stages, including upstream processes like ideation, validation, budgeting, requirements engineering, and specification. This comprehensive approach is vital in reducing the overall time from idea to impact. There are a few steps that can be taken to achieve this.

First, automate repetitive and time-consuming tasks, particularly in testing and deployment. Automating these processes not only accelerates the development cycle but also enhances accuracy and reduces human error. CI/CD pipelines exemplify this approach, enabling teams to integrate and deploy their work frequently and reliably.

Second, look for inefficiencies and delays upstream of the actual software development. Streamlining starts with improving the ideation and validation phases, when agile methodologies like the Lean Startup[1] can be instrumental. Approaches like this advocate for rapid prototyping and iterative testing, ensuring that only validated ideas proceed to the development stage.

Third, pay attention to budgeting and requirements engineering. Traditional annual budget cycles and lengthy requirements gathering phases can significantly delay project initiation. Introducing more agile budgeting practices that allocate funds based on evolving priorities and moving requirements gathering into the actual software development phase can drastically reduce lead times.

Finally, simplifying the specification process, possibly by empowering development teams to take a more active role in solution design, can further reduce bottlenecks. By implementing these strategies, enterprises can achieve a more seamless and efficient software life cycle, from the initial idea to its final deployment and beyond, significantly improving their agility and responsiveness to market needs.

The Balance of Autonomy: Navigating Between Autonomy and Anarchy

AWS Sr. Principal Evangelist Gregor Hope once wrote, “Everyone doing what they think is best isn’t autonomy. That’s anarchy.”

The key to boosting an organization’s performance and speed lies in granting teams autonomy and ensuring they operate independently. Teams having the freedom to make their own decisions leads to shorter cycle times. This increases performance and enhances the overall efficiency of the organization.

However, there’s a limit to how beneficial autonomy can be. Beyond a certain point, teams might continue to improve, but the organization’s overall performance can suffer—particularly from the customer’s perspective.

One example of excessive autonomy is when teams can select their own technology tools and programming languages. This choice significantly affects how teams collaborate and share code. For instance, if two teams use the same programming language, they can share code more easily through methods like pull requests or even collective code ownership. This allows for a flexible degree of interaction. On the other hand, if teams choose different programming languages, it forces a rigid, highly dependent way of working together.

It is critical to embrace local autonomy while balancing it with alignment to untangle the organizational hairball and increase organizational agility and speed. Moving to microservices and optimizing the entire software development life cycle breaks technical and organizational dependencies and drives rapid innovation. This strategy ensures competitive advantage by significantly accelerating the software development process and, as a result, market responsiveness.

[1] Blank, Steve. “Why the Lean Start-Up Changes Everything.” Harvard Business Review, January 3, 2022. https://hbr.org/2013/05/why-the-lean-start-up-changes-everything.