Random Acts of Architecture

Tales of an architect trying to bring order to the chaos that is modern information technology.

Tag Archives: Design

Architect/Stakeholder Inversion

Stakeholder Architect InversionArchitect/stakeholder inversion occurs when non-technical stakeholders tell software architects how a system should work, not what it should do. Without the “what”, software architects are left trying to guess or reverse engineer it. The resulting system may not solve the customer problem or may bloat with features attempting to do so.

Architect/stakeholder inversion is not a stakeholder wanting to move a system into the cloud to reduce costs. It is not wanting a mobile app to reach a different, younger market or offer a better user experience. It is not marketing pushing for a better analytics tool. They have business justifications.

Architect/stakeholder inversion is wanting two products integrated without saying what data to share or tasks to provide. It is creating a report engine without knowing the reports it will run. It is any framework created solely to handle nebulous requirements.

Architect/stakeholder inversion occurs due to one of three reasons. First, non-technical stakeholders feel they need to give low-level, technical requirements. Usually a sign of inexperience or frustration, the stakeholder bypasses discussion with technical details.

Alternatively, software developers may be used to implementing what they are told. This is common in environments with many ancillary roles (user experience, visual design, business analysis, copy writing, solution architecture, application architecture, agile coach, project manager, scrum master, team leader, etc) and stakeholders may take advantage of this.

Second, stakeholders often make technical assumptions and present those assumptions as solutions. They may not even realize they made assumptions.

Technical people may miss the business impacts of technical choices. However, non-technical stakeholders may miss technical impacts of business choices, too. For example, while the ongoing costs of moving to an “Infrastructure as a Service” (IaaS) or “Platform as a Service” (PaaS) provider may be lower, non-technical stakeholders may not consider the transition cost and impacts on compliance, security, jurisdiction, privacy, bandwidth and latency. The stakeholder might not have considered other benefits, such as elasticity (rapid scale up or scale down), built-in monitoring and management tools and cheap creation of test and staging environments, either.

Stakeholders with technical backgrounds may exacerbate the problem. While the technical solution requested may be good, the business context is still needed. Software architects are part of the checks and balances for the business requirements and stakeholder technical knowledge does not negate this.

Third, stakeholders may not yet know the business goals of the system. This may be driven by schedule (“We need to start coding now so that we will hit the deadline”), a misunderstanding of agile processes (“We will work it out as we go”) or a lack of preparation.

Architect/stakeholder inversion is usually solved by highlighting assumptions or providing alternate solutions. Forming these into questions (“Have we considered doing X instead of Y?”) and prototypes/spikes are effective. However, if software architects are on a “need to know” basis, stakeholders set direction solely by intuition instead of evidence or stakeholders take offence at challenges or questions, there may be wider organizational problems.

Architects and stakeholders should cooperate and respectfully challenge each other, providing greater understanding to both sides. Software architects can make better informed design decisions and glean insight into wider and future direction. The stakeholder can get a better understanding of and confidence in the solution.

That said, there are no sides here. Both the stakeholders and architects are working toward the same goal. If the organization has appointed stakeholders and architects, it realizes the value of each. Architect/stakeholder inversion contradicts this and produces a lower quality product.

Update: This post is featured in a discussion in the International Association of Software Architects (IASA) group on Linkedin.

Big Design Up Front versus Emergent Design

BDUF vs Emergent Design(This post is in response to Hayim Makabee’s posts on emergent design and adaptable design along with some of the follow-up discussions, such as the thread in the “97 Things Every Software Architect Should Know” Linkedin Group and Gene Hughson’s post on emergence vs evolution.)

One argument software architects regularly encounter is that time spent designing systems is wasted. Many say that “big design up front” is not the agile way and “emergent design” is more effective. This cuts straight to the value proposition of an architect. If up front design has no place in the Agile world, are architects redundant?

To most people, “big design up front” (BDUF), sometimes called “big up front design” (BUFD), means a lengthy, detailed design created at the start of a project. It works on three assumptions. First, one can create requirements for a project. Second, one can create a design to meet those requirements. Third, the design’s suitability for meeting the requirements can be evaluated without implementing it. In other words, there can be good designs and bad designs.

Meanwhile, emergent design means minimal or no design up front (NDUF). It works on the assumption both the requirements and design must be deduced so the team starts developing the product and iterates as they learn more about the problem and the solution. The process finishes at a predetermined time or when “good enough” requirements and design “emerge”.

By inference, emergent design assumes designs are often highly problem/solution specific. Adapting existing designs may create more work than they save. It also removes the focus from providing value to following the design.

Emergent design is quite popular among Agile and Lean practitioners. They argue emergent design reduces some waste (unnecessary work) by not creating lengthy documents that people may never read. Of the read design documents, few are updated as changes are made. Many developers are so cynical they refuse to read documentation and jump straight to the code to answer questions.

Big design up front may encourage over design. Unnecessary features may be added (violating the YAGNI principle) or the system may be unnecessarily complex (violating the KISS principal). Emergent design, particularly when coupled with Test Driven Design (TDD), can produce the minimum code required to meet a requirement and no more.

Big design up front may create an illusion the team knows more than they do. This may prompt decisions when the team knows the least about the problem, meaning big design up front can become big commitment up front. Meanwhile, a team that delays making decisions until necessary may discover different features are needed.

Big design up front’s assumptions are also not always true. Every project has a goal but it may not be clear how to get there. Most startups do not have quantifiable requirements, for example, where coding is more experimenting than implementing. New technologies may supersede old techniques or require new ones, meaning designs are either too difficult to create or cannot be evaluated without implementation.

However, proponents of big design up front point out that designing is often more useful than design documents. The design exercise validates and challenges requirements, explores edge cases and discovers mistakes. Without it, developers often dive straight into low level details and even a short time thinking about the problem can expose assumptions or alternate solutions they would otherwise miss.

Emergent design assumes change is cheap. A lot of effort and attention has been directed to this. Continuous integration and continuous delivery aim to make releasing easy. Test Driven Design (TDD) and automated testing aim to find regressions quickly. Agile methodologies like Scrum provide visibility and guidance on how to manage change.

However, not every change is cheap. Hardware can be difficult or impossible to change once manufactured. Network infrastructure changes need to be scheduled to minimize impact to others. Engaging external vendors may require lengthy contract negotiations. Legacy code may lack sufficient automated unit test coverage. Aspects like security, compliance and scalability are difficult to retrofit.

Similarly, software development must be accountable to the organization. Required skills and teams must be hired or contracted. Budgets must be determined. Progress is usually tracked against milestones and must be approved based on return on investment (ROI) estimations. Risks must be identified and mitigated. Early designs (as part of architectures) can help drive all of this.

Designs present abstracted views of the system, emphasizing important decisions and removing noise. This means designs can also be reviewed by others before the more expensive and time consuming implementation to find weaknesses or suggest improvements. Designs of notable projects can teach others, either by following or avoiding them.

The problem with comparing big design up front against emergent design is it usually devolves into straw man arguments. Neither are absolutes. Good big design up front recognizes some design and details are filled in during development. Good emergent design must start with some idea of how the system will work.

Both big design up front and emergent design can be done badly. Poor big design up front can miss important factors, provide a poor solution or communicate good ideas badly. Poor emergent design can waste time rewriting code, introduce regressions and impede governance. Both can create a big ball of mud. However, big design up front need not be change averse. Emergent design need not be chaotic and unpredictable.

Big design up front and emergent design are process agnostic. Big design up front originated in waterfall processes. As mentioned above, emergent design is common with agile development methodologies. However, emergent design can be used within a waterfall design phase (prototyping) or for defined components during development (spikes). A team using agile development methodologies may do some design inside, outside or between iterations.

Both approaches can be combined. For example, adaptable design is a technique where parts of the system that anticipate change,such as unknown or changing requirements, are designed to accommodate them.

Looking at the comparison from a different angle, what does “design” mean? Is it thinking about how to approach the system or is it documenting and communicating it? A small system may be something a developer can completely understand and describe in a few sentences. It has an implicit, undocumented design and can be iterated over time using emergent design. However, a large or complex system using a mix of legacy and new components whose development is split across different teams may need a different approach. In other words, the benefit of up front design increases as the system complexity increases.

Different approaches require different skills. Big design up front requires thinking about a system in abstract terms. It is skill that not every developer has, requiring breadth rather than depth, and is often why democratizing design fails. By contrast, emergent design embraces a detail and code focus, particularly with the focus on unit testing and small, incremental changes. This is one reason emergent design is more attractive to software developers.

Both big design up front and emergent design are tools a software development team can use. Rather than being excluded, software architects are in a unique position. They can help determine which approach is best for a situation. The real challenge for a software architect is knowing the right amount of design for a system and when to do it.

An Architect’s Place in Agile

Scrum, the most common implementation of the Agile development methodology, has many well-defined roles. Those that contribute directly to the sprint (a unit of work usually lasting 2-4 weeks) are called “pigs”. Those that consult or assist only are “chickens”, the “scrum master” coordinates the sprint and the “product owner” prioritizes work and ensures the customer needs are met.

So where does the software architect fit in? The architect is not a pig if he or she does not write production code. Is he or she a chicken? The architect needs to be driving his or her features in the sprint and be more involved than a chicken. The architect is not responsible for team organization and a customer representative is usually the product owner.

Going back to basics, why is a software architect needed? Architects are rarely needed in projects with small, co-located teams full of senior developers working on well-defined requirements or well-understood problems. They can usually design and cooperate well enough to produce the desired results. However, large, distributed teams full of junior developers working on vague requirements or complex problems need coordination and direction. This is where architects are most useful.

One way of looking at it is Scrum is a software development methodology, not a productization methodology. Software development is one part of producing a product but there are many other parts, particularly for commercially sold software, such as business case design, marketing, licensing, documentation and localization. The architect could deliver non-functional requirements and high-level designs outside sprints like the other non-development tasks.

However, the architect need not deliver a monolithic document for the high-level design. In keeping with the Agile manifesto, as well as the Lean principle of making decisions as late as possible, the architect only needs to produce enough of the design to unblock the next sprint. The architect will still need a high-level design and identify non-functional requirements initially but Agile recognizes that design is as much a process as a product. Designs for subsequent sprints can be fleshed out in parallel with the development team, minimizing design rework as the team learns more about the problem and finds better solutions.

Could a software architect use Scrum to create the high-level design, either separate to or in parallel with the development teams? This can work if the architect has easy access to the resources he or she needs, such as customers to help understand the business problems, architects from other teams to discuss integration and development managers to check resource estimates. This cannot be guaranteed, particularly with larger, distributed groups – the cases where architects are most useful. However, it will occur in practice if the architect is providing designs for the start of each sprint.

Indeed, if the product owner is remote or often unavailable, an architect fits best into Scrum as a stand-in product owner. This breaks the Scrum rules of only having one product owner. However, different time zones, large projects and multiple commitments mean a single person cannot scale, as a former colleague of mine explained.

Development management may baulk at the perceived loss of control by making an architect a product owner. However, the word “owner” in “product owner” does not mean control of the product, merely creating, prioritizing and clarifying tasks, which architects often do anyway. Architects may not be customers but are judged whether the product meets the requirements or creates business value, just like product owners. They also know the product strategy and have spent time with the customer understanding the problem so are well-suited for this role, using their judgment to determine whether to escalate each question to the product owner.

Moreover, I think the question is not “Where does the architect fit into Agile?”, it’s “How can architects leverage Agile to better perform their role?”. For example, the architect can gain more visibility into the development team’s progress and status (through the backlog and burn down charts). The architect can present the design and gain consensus at the planning meeting that starts a sprint and (hopefully) see it working in the hand-over meeting at the end of a sprint.

Most importantly, architects must be in control of their performance rather than victims of process. A lot of smart people have worked very hard on Agile and Scrum and developers new to Scrum are advised to follow it as written, at least initially, because the reasons behind its nuances are often unclear. However, no development methodology can handle every case, and software architects are one of those things that can fall into the gaps.

%d bloggers like this: