Random Acts of Architecture

Tales of an architect trying to bring order to the chaos that is modern information technology.

Monthly Archives: August 2012

In Defense of Documentation

Anyone who has spent a modicum of time working with software developers learns that developers hate creating designs, use cases, “readmes”, code samples and similar technical documentation. Agile proponents point to the “working software over comprehensive documentation” principle and Lean proponents claim documentation is “waste” since it does not directly contribute to the finished product or customer value.

They have a point. Writing clear, understandable documents is hard and it takes time away from coding. Keeping them up-to-date when inevitable changes occur is similarly time-consuming, particularly large changes or redesigns. Consequently, many stop asking for documentation, exacerbating poor or absent documentation. Many developers argue “code is documentation”. It is always up to date and developers had to work to understand it. Why shouldn’t everyone else?

However, saying “code is the only documentation” is a fallacy. Well written and designed code and indicative automated tests can be good documentation but every project includes code the developers are not proud of. Software developers often think of themselves coding like athletes lining up for a race at the Olympics. They have spent years training and are lining up to compete under perfect conditions. However, frequently code is written when developers are tired, stressed or unfamiliar with the problem or tools and this produces less correct and readable code.

Also, and arguably more important, developers are not the only audience for technical documentation. QA (testers), localization, management and documentation writers (those that write the manuals and non-technical documentation) all need technical information about the product and usually cannot read code, at least not well enough to get the information they need time efficiently. Developers new to the product also need a starting point.

Part of the problem is that documentation, particularly in larger organizations, consists of word processed documents or E-mail. E-mail in particular is a poor choice since it is only available to the recipients and often gets lost. An internal Wiki is a better choice, since most maintain a change history, have centralized storage making the latest version easier to find, support access control, provide auditing and can notify users of changes. Wikis are not perfect, however, since their storage formats are often proprietary and they often lack features of modern word pressing tools.

Wikis also encourage splitting documentation from large, monolithic documents into smaller sections. Care must be taken to ensure consistency and information is easy to find but most Wikis support free text searches. This fits better into the Agile and Lean methodologies, where smaller documents are delivered when needed rather than complete documentation delivered up front.

Writing documentation forces developers to identify and empathize with their customers or other developers, encouraging them to simplify the product’s design, user interface, installation and/or configuration. Some techniques, such as Documentation Drive Development and Readme Driven Development, encourage starting from the documentation for this reason.

It seems that literacy has become a skill in the software development industry. We have fooled ourselves into thinking that developers are only productive if they are writing code and poor documentation is acceptable. As Albert Einstein said “If you can’t explain it simply, you don’t understand it well enough.” Writing documentation forces developers out of their warm, cosy technical silos and back into the user world, reaquainting themselves with the problems the software is trying to solve.

An Architect’s Place in Agile

Scrum, the most common implementation of the Agile development methodology, has many well-defined roles. Those that contribute directly to the sprint (a unit of work usually lasting 2-4 weeks) are called “pigs”. Those that consult or assist only are “chickens”, the “scrum master” coordinates the sprint and the “product owner” prioritizes work and ensures the customer needs are met.

So where does the software architect fit in? The architect is not a pig if he or she does not write production code. Is he or she a chicken? The architect needs to be driving his or her features in the sprint and be more involved than a chicken. The architect is not responsible for team organization and a customer representative is usually the product owner.

Going back to basics, why is a software architect needed? Architects are rarely needed in projects with small, co-located teams full of senior developers working on well-defined requirements or well-understood problems. They can usually design and cooperate well enough to produce the desired results. However, large, distributed teams full of junior developers working on vague requirements or complex problems need coordination and direction. This is where architects are most useful.

One way of looking at it is Scrum is a software development methodology, not a productization methodology. Software development is one part of producing a product but there are many other parts, particularly for commercially sold software, such as business case design, marketing, licensing, documentation and localization. The architect could deliver non-functional requirements and high-level designs outside sprints like the other non-development tasks.

However, the architect need not deliver a monolithic document for the high-level design. In keeping with the Agile manifesto, as well as the Lean principle of making decisions as late as possible, the architect only needs to produce enough of the design to unblock the next sprint. The architect will still need a high-level design and identify non-functional requirements initially but Agile recognizes that design is as much a process as a product. Designs for subsequent sprints can be fleshed out in parallel with the development team, minimizing design rework as the team learns more about the problem and finds better solutions.

Could a software architect use Scrum to create the high-level design, either separate to or in parallel with the development teams? This can work if the architect has easy access to the resources he or she needs, such as customers to help understand the business problems, architects from other teams to discuss integration and development managers to check resource estimates. This cannot be guaranteed, particularly with larger, distributed groups – the cases where architects are most useful. However, it will occur in practice if the architect is providing designs for the start of each sprint.

Indeed, if the product owner is remote or often unavailable, an architect fits best into Scrum as a stand-in product owner. This breaks the Scrum rules of only having one product owner. However, different time zones, large projects and multiple commitments mean a single person cannot scale, as a former colleague of mine explained.

Development management may baulk at the perceived loss of control by making an architect a product owner. However, the word “owner” in “product owner” does not mean control of the product, merely creating, prioritizing and clarifying tasks, which architects often do anyway. Architects may not be customers but are judged whether the product meets the requirements or creates business value, just like product owners. They also know the product strategy and have spent time with the customer understanding the problem so are well-suited for this role, using their judgment to determine whether to escalate each question to the product owner.

Moreover, I think the question is not “Where does the architect fit into Agile?”, it’s “How can architects leverage Agile to better perform their role?”. For example, the architect can gain more visibility into the development team’s progress and status (through the backlog and burn down charts). The architect can present the design and gain consensus at the planning meeting that starts a sprint and (hopefully) see it working in the hand-over meeting at the end of a sprint.

Most importantly, architects must be in control of their performance rather than victims of process. A lot of smart people have worked very hard on Agile and Scrum and developers new to Scrum are advised to follow it as written, at least initially, because the reasons behind its nuances are often unclear. However, no development methodology can handle every case, and software architects are one of those things that can fall into the gaps.

Handling Challenges to Architectural Decisions

One of the most critical aspects of a software architect’s job is the initial high-level design. Based on customer requirements and the architect’s own non-functional requirements, this is where development starts from. It is also written when the development organization knows the least about the problem and so issues often arise as the development team works it.

Challenges are inevitable. It is important is not to panic and not to take it personally. The architect’s goal is to get the best product, not necessarily his or her product. Therefore, first get the other person to explain their issue and reasoning. Common scenarios include:

Different understanding of requirements:  All major analysis and design discussions around the product should involve the architects. (If not, then this indicates possible broader organizational issues.) Explain the decision’s rationale and ask if that matches the other person’s understanding. One or both of you will probably learn something in the ensuing discussion.

Proposes a better solution: Has the person raising the issue tried it before? Did they use a different tool or technique? Do some research to see whether others have had similar problems and their solutions. A small proof of concept is almost always helpful. Non-functional requirements, such as scalability or integration with other systems, are often the deciding factors.

Use an existing solution: This is often a difficult issue. If the person raising the issue was involved in the “better” solution, he or she may take this personally. Explain why it was not selected and work through the decision-making process. It is easy for developers (and architects) to apply their previous work beyond its design goals, even when it is no longer appropriate. However, existing solutions may save time and effort, help standardize products and are politically safer choices.

Resourcing or deadlines: The architect should understand the current and near future resource allocations, such as looking at the Scrum backlog, and do his or her own estimate of resource requirements because this is often an excuse given when the development team feels a task is not in their best interests. The important question here is prioritization rather than total work, so the architect should work with the customer to deprioritize tasks if needed.

Non-technical reasons: Sometimes decisions have to be made quickly, such as to meet changing market requirements or an unexpected opportunity, or people have conditions imposed upon them, such as head count reassignments or reductions. Those in senior positions do not have to justify their decisions but an explanation (usually discretely from a trusted source) may be invaluable. Politics is a fact of life and organizational awareness is important for architects, even if they choose not to “play the game”.

Before considering a solution, what is the impact of the change? Are other components affected? If so, do those teams have the bandwidth to make the changes? Also, does the change use a familiar, proven technology or is it speculative. Each organization has it’s own level of risk tolerance. Clearly, the larger or riskier a change, the more likely to go through a formal change control process. Assuming it is not clearly in that area and a change is warranted, solutions include:

Accept Change: You are only one person and there are always people smarter than you, people more familiar with the problem than you or just luckier. Assuming the impact is acceptable, make the change, communicate it and move on. In the long run, admitting mistakes will earn you more respect than defending the indefensible. If the change is small or low risk, even if it is a worse solution, it may be easier just to accept the change and move on because it gives others a more personal investment in the design.

Compromise: The solution is what neither of you are proposing or bits of both. Although almost cliche, finding a solution that keeps the important aspects of the original design and the person raising the issue happy is best. The architect must be open-minded even if others are not.

Delay: Sometimes the issue raised requires a change that is simply too large to accomplish with the remaining time and resources. In that case, add it to the requirements for the next version.

Escalate to Customer: If there is a disagreement about customer requirements, often the best solution is to take the question to the customer, such as the Scrum product owner. Remember to put this in terms the customer can understand, emphasizing the impact on users. The area may not have been covered in detail and so require more investigation.

Escalate to Management: Perhaps a team is demanding changes that would make their component easier but have a great impact to the rest of the product. Perhaps an external team is demanding changes without an adequate business purpose or return on investment. In this case, it is best to escalate this to the appropriate level of management if no compromise is possible.

Note that infrequent escalation to management or the customer is not a sign of failure. Indeed, involving them in the project can give them much-needed updates and let them know problem areas.

Lastly, do not forget to capture and communicate details of any changes. This may be a short announcement at the next team conference call, entering something into a change log and/or updating the original high level design documents.

No matter how thoroughly an architect researches and presents the high-level design, there will always be those who disagree. Sometimes, it is important to distribute the high-level design to air decisions and elicit feedback. Sometimes, deadlines dictate a incomplete design must be used. Sometimes, someone just wants to prove they know better than the architect. Remember that “architect” is a role, not a rank, and reacting well to challenges may speak more to an architect’s ability than the architecture itself.

%d bloggers like this: