Random Acts of Architecture

Tales of an architect trying to bring order to the chaos that is modern information technology.

Tag Archives: Process

Big Design Up Front versus Emergent Design

BDUF vs Emergent Design(This post is in response to Hayim Makabee’s posts on emergent design and adaptable design along with some of the follow-up discussions, such as the thread in the “97 Things Every Software Architect Should Know” Linkedin Group and Gene Hughson’s post on emergence vs evolution.)

One argument software architects regularly encounter is that time spent designing systems is wasted. Many say that “big design up front” is not the agile way and “emergent design” is more effective. This cuts straight to the value proposition of an architect. If up front design has no place in the Agile world, are architects redundant?

To most people, “big design up front” (BDUF), sometimes called “big up front design” (BUFD), means a lengthy, detailed design created at the start of a project. It works on three assumptions. First, one can create requirements for a project. Second, one can create a design to meet those requirements. Third, the design’s suitability for meeting the requirements can be evaluated without implementing it. In other words, there can be good designs and bad designs.

Meanwhile, emergent design means minimal or no design up front (NDUF). It works on the assumption both the requirements and design must be deduced so the team starts developing the product and iterates as they learn more about the problem and the solution. The process finishes at a predetermined time or when “good enough” requirements and design “emerge”.

By inference, emergent design assumes designs are often highly problem/solution specific. Adapting existing designs may create more work than they save. It also removes the focus from providing value to following the design.

Emergent design is quite popular among Agile and Lean practitioners. They argue emergent design reduces some waste (unnecessary work) by not creating lengthy documents that people may never read. Of the read design documents, few are updated as changes are made. Many developers are so cynical they refuse to read documentation and jump straight to the code to answer questions.

Big design up front may encourage over design. Unnecessary features may be added (violating the YAGNI principle) or the system may be unnecessarily complex (violating the KISS principal). Emergent design, particularly when coupled with Test Driven Design (TDD), can produce the minimum code required to meet a requirement and no more.

Big design up front may create an illusion the team knows more than they do. This may prompt decisions when the team knows the least about the problem, meaning big design up front can become big commitment up front. Meanwhile, a team that delays making decisions until necessary may discover different features are needed.

Big design up front’s assumptions are also not always true. Every project has a goal but it may not be clear how to get there. Most startups do not have quantifiable requirements, for example, where coding is more experimenting than implementing. New technologies may supersede old techniques or require new ones, meaning designs are either too difficult to create or cannot be evaluated without implementation.

However, proponents of big design up front point out that designing is often more useful than design documents. The design exercise validates and challenges requirements, explores edge cases and discovers mistakes. Without it, developers often dive straight into low level details and even a short time thinking about the problem can expose assumptions or alternate solutions they would otherwise miss.

Emergent design assumes change is cheap. A lot of effort and attention has been directed to this. Continuous integration and continuous delivery aim to make releasing easy. Test Driven Design (TDD) and automated testing aim to find regressions quickly. Agile methodologies like Scrum provide visibility and guidance on how to manage change.

However, not every change is cheap. Hardware can be difficult or impossible to change once manufactured. Network infrastructure changes need to be scheduled to minimize impact to others. Engaging external vendors may require lengthy contract negotiations. Legacy code may lack sufficient automated unit test coverage. Aspects like security, compliance and scalability are difficult to retrofit.

Similarly, software development must be accountable to the organization. Required skills and teams must be hired or contracted. Budgets must be determined. Progress is usually tracked against milestones and must be approved based on return on investment (ROI) estimations. Risks must be identified and mitigated. Early designs (as part of architectures) can help drive all of this.

Designs present abstracted views of the system, emphasizing important decisions and removing noise. This means designs can also be reviewed by others before the more expensive and time consuming implementation to find weaknesses or suggest improvements. Designs of notable projects can teach others, either by following or avoiding them.

The problem with comparing big design up front against emergent design is it usually devolves into straw man arguments. Neither are absolutes. Good big design up front recognizes some design and details are filled in during development. Good emergent design must start with some idea of how the system will work.

Both big design up front and emergent design can be done badly. Poor big design up front can miss important factors, provide a poor solution or communicate good ideas badly. Poor emergent design can waste time rewriting code, introduce regressions and impede governance. Both can create a big ball of mud. However, big design up front need not be change averse. Emergent design need not be chaotic and unpredictable.

Big design up front and emergent design are process agnostic. Big design up front originated in waterfall processes. As mentioned above, emergent design is common with agile development methodologies. However, emergent design can be used within a waterfall design phase (prototyping) or for defined components during development (spikes). A team using agile development methodologies may do some design inside, outside or between iterations.

Both approaches can be combined. For example, adaptable design is a technique where parts of the system that anticipate change,such as unknown or changing requirements, are designed to accommodate them.

Looking at the comparison from a different angle, what does “design” mean? Is it thinking about how to approach the system or is it documenting and communicating it? A small system may be something a developer can completely understand and describe in a few sentences. It has an implicit, undocumented design and can be iterated over time using emergent design. However, a large or complex system using a mix of legacy and new components whose development is split across different teams may need a different approach. In other words, the benefit of up front design increases as the system complexity increases.

Different approaches require different skills. Big design up front requires thinking about a system in abstract terms. It is skill that not every developer has, requiring breadth rather than depth, and is often why democratizing design fails. By contrast, emergent design embraces a detail and code focus, particularly with the focus on unit testing and small, incremental changes. This is one reason emergent design is more attractive to software developers.

Both big design up front and emergent design are tools a software development team can use. Rather than being excluded, software architects are in a unique position. They can help determine which approach is best for a situation. The real challenge for a software architect is knowing the right amount of design for a system and when to do it.

Should Software Architects Write Code?

Much has been written and debated on whether software architects should write code. Many argue the more architects understand the language, tools and environment they are designing for, the more effective they are and this is best achieved by implementing some or all of the design. Non-coding architects, sometimes called “PowerPoint architects”, “astronaut architects” or “ivory tower architects”, may use archibabble and talkitecture to convince non-technical stakeholders of their expertise while delegating the unsolved, real problems to developers, so much so that it has become an organizational pattern (“Architect Also Implements“) and corresponding anti-pattern (“Architects Don’t Code“). Others argue that architects responsible for implementing their architectures lose focus on the bigger issues and longer term vision. Understanding does not necessarily require knowledge of the minutiae and, as systems scale up and diversify, implementing it requires too much time or spreads the architect too thin. Therefore, should software architects write code?

As with many difficult questions, the problem starts with the question itself. “Should a software architect write code?” can mean “Should a software architect always prototype or implement their own architectures?”, “Should a software architect write production code most of the time?” or “Should a software architect be able to write code?”. It could also mean “Is coding the best or only way to become a software architect?” or “Can non-coders be good architects?” but that is best left to another blog post.

It also depends on the definition of “software architect”. The Canadian architect (of buildings rather than IT) Witold Rybczynski wrote in his 1989 book “The Most Beautiful House in the World“:

“For centuries, the difference between master masons, journeymen builders, joiners, dilettantes, gifted amateurs, and architects has been ill defined. The great Renaissance buildings, for example, were designed by a variety of non-architects. Brunelleschi was trained as a goldsmith; Michelango as a sculptor, Leonardo da Vinci as a painter, and Alberti as a lawyer; only Bramante, who was also a painter, had formally studied building. These men are termed architects because, among other things, they created architecture — a tautology that explains nothing.”

This is exactly the same issue for software architects. Without a clearly defined and segregated role, anyone designing software or IT related systems can rightly be called an architect, including many developers and technical leads. For the sake of argument, this post uses Simon Brown’s definition, where software architects are responsible for high level design, non-functional requirements and technical vision.

Should a software architect should be able to write code? Architects should be able to read and write code because it:

  1. Verifies the code written by developers matches the design and identifies deviations.
  2. Helps the architect learn about changes or new features. If the architect has been assigned to a new project, he or she can learn the product sooner by looking at the code, too.
  3. Allows the architect to write a proof of concepts or prototype. A working demo is much more convincing than an architecture diagram and will usually facilitate better estimates. Care must be taken to prevent non-technical stakeholders attaching too much credibility to it, as with any prototype however.
  4. Provides another pair of capable hands during the project crunch periods.
  5. Makes the architect more forgiving of bugs because the architect has likely made similar mistakes in the past. At least, the architect should have a better understanding what types of issues to expect.

Writing code may help earn the architect respect of the developers. Developers can be notoriously dismissive and a software architect producing some of his or her own code, even if it is just a proof of concept, or providing good feedback from a code review can make the developers feel like the architect is one of them. Having a working development environment and access to source code means the architect can try out new versions without waiting for a build or release. Any significant build, development environment or source code control issues also become apparent to the architect.

Note that code reviews do not replace talking to developers because regular discussions between developers and software architects can help build mutual respect. Otherwise, developers may see the architect as a constraint or threat that must be circumvented. Also, developers often know or can find problem areas faster than the architect reading the code but there needs to be a balance between architect self-sufficiency and squandering developer time.

Software architects are often required to settle disputes between developers, such as when one team discovers a better way of solving a problem or that the proposed design will be harder to implement than first thought. Software architects are also sometimes mentors or coaches for developers or may be used as internal consultants to examine process, quality, automation or similar issues. Understanding code means the architect can use his or her judgment more effectively rather than rely on which developer is more persuasive.

Should a software architect write production code most of the time (usually implementing their own architecture)? If a software architect implements their own architectures, this ensures the design is implementable with the tools and environment used. This can lead to new insights, improved designs and more accurate estimations. It is also implemented by the person most familiar with the design so minimizes miscommunication.

However, architects may jump to implementation (depth thinking) before exhausting other solutions (breadth thinking). Existing implementations may overly influence the architect or the architect become attached to his or her code, fighting against needed improvements. It can distract the architect from higher level tasks such as longer term planning, communicating with stakeholders and reviewing other developers’ code.

Also, part of the role of an architect is to fight for reuse, security and other non-functional requirements. Being forced to prototype or implement their design may encourage compromises that the team need not make. It is not that an architect does not make compromises – design is the art of compromise as many have said – more that it is the architect’s job to make the right compromises rather than those the architect makes creating the initial implementation. The developers will likely rewrite much of the architect’s code, anyway.

Indeed, the more an architect focuses on communication, requirements analysis, stakeholder management and non-technical activities, the more the development skills of software architects may atrophy. As long as the architect is providing value via other means this is not an issue. However, an architect should maintain his or her development skills; whether by extensive research, working on their own projects or contributing where they can; but needs to focus on capabilities, limitations and edge cases rather than speed or a complete understanding.

Problems may occur when organizations promote their strongest developers into a software architect role rather than good communicators that are capable of working at higher levels of abstraction. Friction arises when these architects try to “lead from the front” by implementing their architecture rather than facilitating others to do so. Organizations should promote a developer that has better soft skills, instead.

Many confuse not writing code with a lack of feedback. An unprototyped architecture, hypothetically, may be difficult to implement or problematic. However, a senior developer or technical lead can prototype the architecture if required. This also allows architects and developers to work together and ensure the design is communicated well. Alternatively, the architecture can be shared with others that have implemented similar systems previously or architects or developers working on integrating products. Requiring an architect to implement their own architecture beyond a proof of concept also does not scale well, particularly for large or complex products.

Similarly, many confuse an architect not writing code with a lack of accountability. Architects must produce designs that not only are approved (whether formally or informally) by stakeholders but also developers and developers should not approve a document that does not meet their needs. Issues or errors in the designs should be noted. Some change is expected but major or expensive errors should be attributed to the architect. An architect implementing their architecture in code does not guarantee an issue-free project.

With increased use of agile development methodologies, architects are no longer creating an architecture and “throwing it over the wall” to developers. Even previously ivory tower architects are more involved with lower level issues since less critical decisions are deferred until later in the process and design is iterative. For example, architects in organizations using Scrum should attend at least the planning, review and retrospective meetings. (Some architects may move to other projects or otherwise not see the project through, the “Architects Play Golf” pattern. This is an organizational issue and unrelated to whether architects code.)

Many developers also look down on “PowerPoint architectures”. However, many forget the role of a software architect is as much communication as development and completed, implemented architecture does not help non-technical stakeholders, QA, localization, documentation writers and so on. Of course, these stratospheric PowerPoint architectures are not substitutes for high-level designs developers can implement but the architect represents the developers and products to outsiders and developers often feel any time not spent developing is unproductive. Ultimately, PowerPoint architectures do have their place but developers are as much the architects’ customers as the stakeholders.

Should architects write code? The question is loaded and should be determined by the team on a case by case basis. Architects may prototype high risk projects, experiment with new libraries or try out now tools. Architects may completely delegate the design and implementation of well understood, low risk components. The real question is “How does an architect be successful?”. It is a question of managing and mitigating risk. Architects are often good coders but good coders are not necessarily good architects.

Update: There is a large discussion about this post on the IASA (International Association of Software Architects) LinkedIn group: http://www.linkedin.com/groups/Should-Software-Architects-Code-1523.S.188454845

An Architect’s Place in Agile

Scrum, the most common implementation of the Agile development methodology, has many well-defined roles. Those that contribute directly to the sprint (a unit of work usually lasting 2-4 weeks) are called “pigs”. Those that consult or assist only are “chickens”, the “scrum master” coordinates the sprint and the “product owner” prioritizes work and ensures the customer needs are met.

So where does the software architect fit in? The architect is not a pig if he or she does not write production code. Is he or she a chicken? The architect needs to be driving his or her features in the sprint and be more involved than a chicken. The architect is not responsible for team organization and a customer representative is usually the product owner.

Going back to basics, why is a software architect needed? Architects are rarely needed in projects with small, co-located teams full of senior developers working on well-defined requirements or well-understood problems. They can usually design and cooperate well enough to produce the desired results. However, large, distributed teams full of junior developers working on vague requirements or complex problems need coordination and direction. This is where architects are most useful.

One way of looking at it is Scrum is a software development methodology, not a productization methodology. Software development is one part of producing a product but there are many other parts, particularly for commercially sold software, such as business case design, marketing, licensing, documentation and localization. The architect could deliver non-functional requirements and high-level designs outside sprints like the other non-development tasks.

However, the architect need not deliver a monolithic document for the high-level design. In keeping with the Agile manifesto, as well as the Lean principle of making decisions as late as possible, the architect only needs to produce enough of the design to unblock the next sprint. The architect will still need a high-level design and identify non-functional requirements initially but Agile recognizes that design is as much a process as a product. Designs for subsequent sprints can be fleshed out in parallel with the development team, minimizing design rework as the team learns more about the problem and finds better solutions.

Could a software architect use Scrum to create the high-level design, either separate to or in parallel with the development teams? This can work if the architect has easy access to the resources he or she needs, such as customers to help understand the business problems, architects from other teams to discuss integration and development managers to check resource estimates. This cannot be guaranteed, particularly with larger, distributed groups – the cases where architects are most useful. However, it will occur in practice if the architect is providing designs for the start of each sprint.

Indeed, if the product owner is remote or often unavailable, an architect fits best into Scrum as a stand-in product owner. This breaks the Scrum rules of only having one product owner. However, different time zones, large projects and multiple commitments mean a single person cannot scale, as a former colleague of mine explained.

Development management may baulk at the perceived loss of control by making an architect a product owner. However, the word “owner” in “product owner” does not mean control of the product, merely creating, prioritizing and clarifying tasks, which architects often do anyway. Architects may not be customers but are judged whether the product meets the requirements or creates business value, just like product owners. They also know the product strategy and have spent time with the customer understanding the problem so are well-suited for this role, using their judgment to determine whether to escalate each question to the product owner.

Moreover, I think the question is not “Where does the architect fit into Agile?”, it’s “How can architects leverage Agile to better perform their role?”. For example, the architect can gain more visibility into the development team’s progress and status (through the backlog and burn down charts). The architect can present the design and gain consensus at the planning meeting that starts a sprint and (hopefully) see it working in the hand-over meeting at the end of a sprint.

Most importantly, architects must be in control of their performance rather than victims of process. A lot of smart people have worked very hard on Agile and Scrum and developers new to Scrum are advised to follow it as written, at least initially, because the reasons behind its nuances are often unclear. However, no development methodology can handle every case, and software architects are one of those things that can fall into the gaps.

%d bloggers like this: