Random Acts of Architecture

Wisdom for the IT professional, focusing on chaos that is IT systems and architecture.

Intellectual Property Ownership

Whenever anyone involved in intellectual property starts a new job, particularly in software development, the employment contract usually includes a lengthy intellectual property agreement where the new hire assigns ownership of all intellectual property created over to the employer. Many new employees balk at this, concerned that the employer will claim any side projects they are working on and with recent court battles over software patents worth billions of dollars, for example, intellectual property ownership is a complex area worth exploring.

Employment contracts are usually “work for hire” contacts. Although the legal requirements and obligations vary from country to country, it usually means the employer owns any works created instead of the employee, in return for payment or salary. Few software developers dispute this. However, unlike authors and artists usually contracted for a specific work, developers are often contracted for all intellectual property created during their employment, irrespective of whether it is performed during working hours on work equipment.

Software developers are increasingly working on side projects like open source software or apps for mobile or tablet app stores. Many developers code in the spare time, whether it be tweaking the website for a friend’s business or experimenting with new libraries or languages. It is also increasingly common for students to bolster their resume this way or for software developers to teach themselves new languages or libraries.

This differs from traditional “working under the table” or “moonlighting” in that much of the work is not paid. The software developer may want to capitalize on the work in the future but usually just does not want others intruding or demanding ownership of the work, akin to a lawyer doing “pro bono” work. Software developers are also producing assets other than code. For example, software patents produced during software development can be more valuable than the code.

The barrier of entry for software development has never been lower. Thirty years ago, software development was difficult, usually performed on expensive, centralized computers and proprietary software. This is a far cry from today where one can create complex websites using free tools running on free operating systems hosted on commodity priced servers. Compare that to chemists, physicists or engineers that may require thousands or millions of dollars of equipment and dedicated teams of support staff to perform their research and yet more to monetize it.

Indeed, the problem with software patents and intellectual property ownership in software development is development only part of the cost. There is the marketing and sales required to turn products into revenue, for example, and the IT infrastructure, HR and accounting structures required to support all of this.

Patents are similar. Beyond software development, legal expertise to file patents is required along with the time and resources needed to find and deal with infringements. Patents are also often cross licensed, either earning additional revenue or allowing access to other organizations’ patents. Patents may also become more valuable over time as products they are used in become more widespread.

The word “ownership” also carries many mis- and preconceptions. If an employed software developer (“inventor”) wants to own all or part of his or her inventions, what does this mean? Does the inventor want royalties, like an actor may receive for a movie? Does the inventor want the option to use it in their own work, possibly for a job at a competitor? What about an open source project the inventor contributes to in his or her spare time? Does the inventor want the option to stop others using it, like competitors or those the inventor disagrees with? Will the developer help fund the sales, marketing and legal infrastructures required?

Even if those in software development can make a case for increased “ownership” of their products, it is not in employers’ interest to allow this. The creative process cannot be constrained to occur within work hours or on work equipment – many have  inspiration when asleep, exercising or in the shower – and increasingly flexible working arrangements further blur the distinction. Work for hire contracts are also well understood and widespread, making them lower risk.

Some would argue software development is like a painting selling and reselling for increasing amounts but the original painter seeing none of the profit.  However, a better analogy is performers selling music. Do they go through a record label where they get greater exposure and marketing but sacrifice income or do they produce the songs themselves and sell it through iTunes, where they can make a greater cut but a much reduced sales volume?

Many employers are also quite reasonable. If the idea is unrelated to current or likely projects and the employee is not going to make much money, pursuing it is not worth the expense. If the organization files patents, most reward developers with bonuses for doing so. Other employers take the opposite position and talking to the employer first before producing anything important is prudent.

It will be interesting to see what the future holds. Younger generations, those that have grown up with social networking, are used to sharing their lives on social media and regularly blur the lines between professional and social. With software development tools more accessible than ever and collaborative source code repositories like github gaining in popularity, will developers from younger generations look at coding the same way? While they may have different politics from their GNU and GPL espousing forefathers, will they see “social coding” or “social software development” as an obvious direction? If so, what compromises will be made?

Software Development: An Overtime Culture

Software development is rife with overtime. Driven by passion and perfectionism, many developers throw everything into it, reveling in a culture of coding in darkened rooms late at night. Coupled with customers pushing for more or larger deliverables with fewer staff and a legal exemption from overtime pay in many circumstances, developers often shoulder the burden. Unfortunately, consistently working late nights and weekends is not sustainable but many seem unable to break free of the rut.

To put this in perspective, overtime is not unique to software development or even IT – just ask accountants around the end of financial year or salespeople in danger of missing their quota. There is also nothing wrong with flexible working hours and arrangements. Nevertheless, a work/life balance is important particularly for those with families or other commitments outside work.

It is easy to blame overtime on poor management. While true, saying “manage better” does not solve the problem. Indeed, there are many causes of software development overtime, including: (1) poor estimation or underestimation, (2) too few or under-skilled developers, (3) perfectionism, (4) redoing work due to past mistakes or (5) changing requirements/poor communication with other parts of the business. Specifically:

Poor estimation or underestimation: Few cling to the Newtonian beliefs that estimation is merely substituting initial measurements into well understood formulas to predict the delivery date.  The issue is one of negotiation – experienced software developers often know how much features will really cost in time and resources but software developers are often outclassed by customers or management used to arguing their case and, once agreed, deadlines are traditionally immutable, like a salesperson’s quota.

Too few staff or under-skilled developers: No development team is has “enough” developers and with the current trend of outsourcing and offshoring, developers working on project may be under-skilled or unfamiliar with the business problem. However, “you have to go to war with the army you have” as they say but the Dunning-Kruger effect and poor planning rarely take this (and Murphy’s Law) into account.

Perfectionism: Unfortunately, software developers can be their own worst enemies. Well architected, readable, tested software is the goal but “production quality code” is really a tautology – real developers ship. Include time for refactoring but time box it. Include time for design but realize it will change. Include time for testing but test important features first.

Correcting development mistakes: Unfortunately, mistakes happen but software developers must take responsibility for their own mistakes. To put this in context, at least development has the option of working late to fix their mistakes, unlike sales or management.

Changing requirements or poor communication: Many developers think of this as an “us versus them” situation – the “villainous” business “demands” changes and the developers are “helpless victims.” Instead, work on negotiation and trade-offs. After all, if the project is not delivered, everyone suffers, not just development. Make it a business problem or a business risk.

Note the steady addition of new requirements is a smell, possibly indicating the release cycle is too long. If possible, consider multiple smaller releases rather than a single large one.

There is no silver bullet to prevent overtime. Software development is a profession and there will always be work to do. However, it is possible to reduce it. Apart from focusing on soft skills, developers need to:

  1. Improve negotiation skills. For example, treat estimates as immutable. Trade features rather than agree to reduce estimates. Otherwise the estimates lose credibility. Treat everything else (deadlines, delivery mechanisms, support, documentation, maintenance releases and so on)  as negotiable.
  2. Improve estimation. Each development team faces its own challenges in this area but a few suggestions include:
    1. Improve prioritization. Many customers want everything but estimating the revenue or customers for each feature (or loss if not implemented) is usually the best prioritization strategy. Deprioritize features that have no customer impact (the lean software development definition of waste), too.
    2. Estimate with 10-20% fewer resources and less time. Customers often spot padded estimates (reducing their credibility leading them to press for reductions) but reducing the resources and time instead is usually seen as prudent.
    3. Present estimates using the least accurate unit of measure. Instead of saying “172.3 hours”, say “about 1 month”. Defer more detailed estimates until the problem is better understood later in the project.
  3. Track the previous work estimates against actual time and resources. Use that to support current estimates and improve future estimation and negotiation.
  4. Focus on delivering. Many developers love learning new things and striving for the ultimate solution. There is nothing wrong with this but a critical project is usually not the place for it. Temper or time box this experimentation and perfectionism.

Agile development methodologies might also be useful but suggest them tentatively. Many have presented ill-defined “agile development” practices as a panacea, thus some consider the term “agile” a less than credible buzzword. Agile development methodologies require a wider organizational changelike the appointment and recognition of single product owners in scrum, and may require more trust before allowing this.

The challenge with much of this is not sacrificing status of development within the organization or being seen as an impediment. The business may expect overtime and changing this perception can be hard. Metrics that demonstrate how things get worse as overtime increases, like as lines of code produced and bugs introduced, might help particularly against those that espouse the “pressure makes diamonds” fallacy.

Reducing it requires developers to recognize their part in the problem. There is nothing wrong with developers striving to create the best solution and putting in the extra effort when they want to but required and expected overtime is problematic. Ignoring extreme cases like those described by EA_Spouse, the best solution is usually a mutually constructive one, particularly with developers improving their negotiation skills.

Should Software Architects Write Code?

Much has been written and debated on whether software architects should write code. Many argue the more architects understand the language, tools and environment they are designing for, the more effective they are and this is best achieved by implementing some or all of the design. Non-coding architects, sometimes called “PowerPoint architects”, “astronaut architects” or “ivory tower architects”, may use archibabble and talkitecture to convince non-technical stakeholders of their expertise while delegating the unsolved, real problems to developers, so much so that it has become an organizational pattern (“Architect Also Implements“) and corresponding anti-pattern (“Architects Don’t Code“). Others argue that architects responsible for implementing their architectures lose focus on the bigger issues and longer term vision. Understanding does not necessarily require knowledge of the minutiae and, as systems scale up and diversify, implementing it requires too much time or spreads the architect too thin. Therefore, should software architects write code?

As with many difficult questions, the problem starts with the question itself. “Should a software architect write code?” can mean “Should a software architect always prototype or implement their own architectures?”, “Should a software architect write production code most of the time?” or “Should a software architect be able to write code?”. It could also mean “Is coding the best or only way to become a software architect?” or “Can non-coders be good architects?” but that is best left to another blog post.

It also depends on the definition of “software architect”. The Canadian architect (of buildings rather than IT) Witold Rybczynski wrote in his 1989 book “The Most Beautiful House in the World“:

“For centuries, the difference between master masons, journeymen builders, joiners, dilettantes, gifted amateurs, and architects has been ill defined. The great Renaissance buildings, for example, were designed by a variety of non-architects. Brunelleschi was trained as a goldsmith; Michelango as a sculptor, Leonardo da Vinci as a painter, and Alberti as a lawyer; only Bramante, who was also a painter, had formally studied building. These men are termed architects because, among other things, they created architecture — a tautology that explains nothing.”

This is exactly the same issue for software architects. Without a clearly defined and segregated role, anyone designing software or IT related systems can rightly be called an architect, including many developers and technical leads. For the sake of argument, this post uses Simon Brown’s definition, where software architects are responsible for high level design, non-functional requirements and technical vision.

Should a software architect be able to write code? Architects should be able to read and write code because it:

  1. Verifies the code written by developers matches the design and identifies deviations.
  2. Helps the architect learn about changes or new features. If the architect has been assigned to a new project, he or she can learn the product sooner by looking at the code, too.
  3. Allows the architect to write a proof of concepts or prototype. A working demo is much more convincing than an architecture diagram and will usually facilitate better estimates. Care must be taken to prevent non-technical stakeholders attaching too much credibility to it, as with any prototype however.
  4. Provides another pair of capable hands during the project crunch periods.
  5. Makes the architect more forgiving of bugs because the architect has likely made similar mistakes in the past. At least, the architect should have a better understanding what types of issues to expect.

Writing code may help earn the architect respect of the developers. Developers can be notoriously dismissive and a software architect producing some of his or her own code, even if it is just a proof of concept, or providing good feedback from a code review can make the developers feel like the architect is one of them. Having a working development environment and access to source code means the architect can try out new versions without waiting for a build or release. Any significant build, development environment or source code control issues also become apparent to the architect.

Note that code reviews do not replace talking to developers because regular discussions between developers and software architects can help build mutual respect. Otherwise, developers may see the architect as a constraint or threat that must be circumvented. Also, developers often know or can find problem areas faster than the architect reading the code but there needs to be a balance between architect self-sufficiency and squandering developer time.

Software architects are often required to settle disputes between developers, such as when one team discovers a better way of solving a problem or that the proposed design will be harder to implement than first thought. Software architects are also sometimes mentors or coaches for developers or may be used as internal consultants to examine process, quality, automation or similar issues. Understanding code means the architect can use his or her judgment more effectively rather than rely on which developer is more persuasive.

Should a software architect write production code most of the time (usually implementing their own architecture)? If a software architect implements their own architectures, this ensures the design is implementable with the tools and environment used. This can lead to new insights, improved designs and more accurate estimations. It is also implemented by the person most familiar with the design so minimizes miscommunication.

However, architects may jump to implementation (depth thinking) before exhausting other solutions (breadth thinking). Existing implementations may overly influence the architect or the architect become attached to his or her code, fighting against needed improvements. It can distract the architect from higher level tasks such as longer term planning, communicating with stakeholders and reviewing other developers’ code.

Also, part of the role of an architect is to fight for reuse, security and other non-functional requirements. Being forced to prototype or implement their design may encourage compromises that the team need not make. It is not that an architect does not make compromises – design is the art of compromise as many have said – more that it is the architect’s job to make the right compromises rather than those the architect makes creating the initial implementation. The developers will likely rewrite much of the architect’s code, anyway.

Indeed, the more an architect focuses on communication, requirements analysis, stakeholder management and non-technical activities, the more the development skills of software architects may atrophy. As long as the architect is providing value via other means this is not an issue. However, an architect should maintain his or her development skills; whether by extensive research, working on their own projects or contributing where they can; but needs to focus on capabilities, limitations and edge cases rather than speed or a complete understanding.

Problems may occur when organizations promote their strongest developers into a software architect role rather than good communicators that are capable of working at higher levels of abstraction. Friction arises when these architects try to “lead from the front” by implementing their architecture rather than facilitating others to do so. Organizations should promote a developer that has better soft skills, instead.

Many confuse not writing code with a lack of feedback. An unprototyped architecture, hypothetically, may be difficult to implement or problematic. However, a senior developer or technical lead can prototype the architecture if required. This also allows architects and developers to work together and ensure the design is communicated well. Alternatively, the architecture can be shared with others that have implemented similar systems previously or architects or developers working on integrating products. Requiring an architect to implement their own architecture beyond a proof of concept also does not scale well, particularly for large or complex products.

Similarly, many confuse an architect not writing code with a lack of accountability. Architects must produce designs that not only are approved (whether formally or informally) by stakeholders but also developers and developers should not approve a document that does not meet their needs. Issues or errors in the designs should be noted. Some change is expected but major or expensive errors should be attributed to the architect. An architect implementing their architecture in code does not guarantee an issue-free project.

With increased use of agile development methodologies, architects are no longer creating an architecture and “throwing it over the wall” to developers. Even previously ivory tower architects are more involved with lower level issues since less critical decisions are deferred until later in the process and design is iterative. For example, architects in organizations using Scrum should attend at least the planning, review and retrospective meetings. (Some architects may move to other projects or otherwise not see the project through, the “Architects Play Golf” pattern. This is an organizational issue and unrelated to whether architects code.)

Many developers also look down on “PowerPoint architectures”. However, many forget the role of a software architect is as much communication as development and completed, implemented architecture does not help non-technical stakeholders, QA, localization, documentation writers and so on. Of course, these stratospheric PowerPoint architectures are not substitutes for high-level designs developers can implement but the architect represents the developers and products to outsiders and developers often feel any time not spent developing is unproductive. Ultimately, PowerPoint architectures do have their place but developers are as much the architects’ customers as the stakeholders.

Should architects write code? The question is loaded and should be determined by the team on a case by case basis. Architects may prototype high risk projects, experiment with new libraries or try out now tools. Architects may completely delegate the design and implementation of well understood, low risk components. The real question is “How does an architect be successful?”. It is a question of managing and mitigating risk. Architects are often good coders but good coders are not necessarily good architects.

Update: There is a large discussion about this post on the IASA (International Association of Software Architects) LinkedIn group: http://www.linkedin.com/groups/Should-Software-Architects-Code-1523.S.188454845

A Ticket to Security via Obscurity

A group of Australian university students recently “cracked” the “encryption” used by public transport tickets in New South Wales (NSW), Australia. The public transport ticket format’s security relied on people not knowing the format of data on the ticket’s magnetic stripe and not having access to equipment to create tickets. In other words, it relied on “security through obscurity“.

Security through obscurity is not a new concept. In 1883, Auguste Kerckhoff theorised that the only truly secure parts about a system (computerized or otherwise) were the hidden or secret parts. This became known as Kerckhoff’s principle.  Over half a century later, Claude Shannon rephrased it as assume “the enemy knows the system”, called Shannon’s Maxim. These are as true now in computer security, particularly cryptography and encryption,  as they were then.

Ultimately, poor computer security for the public transport ticket system is not the programmers’ or engineers’ fault. The customer and management decided the security provided was sufficient. It is similar to the Year 2000 issue. The programmers knew that two digit years were not going to work after the century changed but why should the manager responsible spend resources to fix it when he or she is unlikely to be around in the year 2000 to take credit for the work?

When people initially approach computer security, they start at cryptography and software security. If we build more secure encryption systems; if we  fix every buffer overflow and every injection attack; if every object has the appropriate authentication, access control, auditing and so on, the “bad guys” will be helpless.

A lot of progress has been made in improving computer security. Organizations like OWASP and SAFECode provide software security guidance for software developers. Many vendors provide frameworks, such as Microsoft’s Security Development Lifecycle, and software patching is a regular occurrence. The US government has FIPS 140-2, a standard for hardware or software that performs cryptography, and Common Criteria is an attempt by multiple governments to provide a framework for software security assurance. A plethora of software security tools are also available with interesting and technical sounding names like “static analysis tools” and “fuzzers”.

That said, computer security practitioners eventually realise it is a matter of economics instead – as Allan Schiffman said in 2004, “amateurs study cryptography; professionals study economics.” It is not that we do not know how to build secure systems. However, if the customer cannot differentiate the security (or quality) of two otherwise seemingly identical products, the cheaper (and likely less secure) product will win. Similarly, most software is sold “as is”, meaning the customer assumes all the risk. See Bruce Schneier’s “Schneier on Security” or “Freakonomics” by Steven D. Livett and Stephen J. Dubner for more discussion.

Others take a different direction and focus on the sociology and behavioural patterns behind security.  Although most people recognise computer security is important, the first reaction of most when confronted with a security warning is to ignore or bypass it. After all, social engineering attacks are behind many of the headline grabbing computer security incidents over the last few years, such as Operation Aurora and Stuxnet. See Bruce Schneier’s “Liars and Outliers” for more discussion.

That said, there is little impetus to improve computer security without people willing to push the boundaries of systems and reveal flaws. Kudos and credit to the students for not only discovering  the public transport ticket format and “encryption” mechanism (if true encryption was actually ever used) but revealing it in a responsible (or “white hat”) manner, such as by allowing the targeted state to reveal itself. Remember that just because the ticket format has been discovered by the students does not mean others have not already done so and used it for less than honest purposes.

Interestingly, the NSW public transport ticket system is approximately 20 years old, introduced when computers were less powerful and less accessible and the required security standards were much lower. I wonder how many systems being built today are going to be considered secure in 20 years time, when computers are going to be presumably exponentially more powerful and more ubiquitous and the security standard much higher.

A Clash of Development Cultures

Software developers often consider themselves perfectly rational. After all, their jobs consist of programming deterministic, rational machines (computers). However, anyone who has worked with developers realises this is completely untrue. In particular, leaders that direct or influence different groups of developers need to understand what motivates them, especially the more vocal and influential developers and technical leads.

There are two types of influential developers. The first are the “craftsmen” (or the gender non-specific “craftspeople”). Craftsmen are motivated by the respect of their peers and want to be known as “good” developers. Craftsmen tend to:

  • Want to work with cutting edge tools, such as the latest version of an IDE or library, and techniques, such as agile development methodologies.
  • Think “good” code not only satisfies the requirements but is clear, maintainable and is an example to others. They are more likely to refactor code, follow (or create) coding  style guidelines and create automated tests, particularly when they have advance warning their code is going to be reviewed.
  • Thoroughly research and experiment with different ways of solving a problem. They are often critical of designs or direction from others, particularly those removed form the code.
  • Prefer a more egalitarian, flatter management and team structure. They are happy to work in a group of peers, each with their own area of responsibility.
  • Code in their spare time, discuss technical issues over lunch and read or write technical blogs. They hold examples of collaboration, such as open source software, in high regard. They tend to look past certifications and degrees to focus on what was studied or produced.
  • Hold technical opinions strongly. They may argue weakly because they feel their points are “obvious”.
  • Be more common in western cultures like in the US, UK and Australia.

The second are “success-oriented“. They see software development as a relatively well paid and respected stepping stone, usually to management.  They tend to:

  • Prefer familiar tools and techniques, so long as they are well compensated and have promotion opportunities.
  • Think “good” code meets requirements. Other facts are less important.
  • Seek help for problems when there is no immediate, obvious solution.
  • Look to seniority or management for guidance and direction. Indeed, if they are in a senior position, they will often demand the right to design or give direction.
  • Respect and seek “authoritative” measures of achievement such as degrees, certifications and training. The goal is usually to decrease the time to the next promotion, enter the industry at a higher position or be paid more.
  • Hold technical opinions weakly. They may argue strongly because it their training and education emphasized it.
  • Be more common in eastern cultures like in India and China.

Clearly, these groups are not well-defined or mutually exclusive and are generalizations. Craftsmen still want promotions and pay rises. Success-oriented developers may still be interested in technology. They key difference is that the software written by craftsmen is part of their identity so they work to make it as good as they can. They grok languages rather than just learn them.

Craftsmen are not better developers than success-oriented developers and vice versa. For example, craftsman may get distracted by perfectionism or experiments but may drive the organization to newer and (hopefully) better processes. On the other hand, success-oriented developers often use well-understood paths to complete tasks sooner and are happy to maintain older software using older tools.

Success-oriented developers are not more likely to provide business value than craftsmen. Although success-oriented developers are less dismissive of non-technical users and focus more on pleasing management, they commonly implement the requirements verbatim without additional understanding.

Although team composition and work assignment are often the domain of team leaders and management, it benefits everyone in the team to understand others’ motivations. For example, pointing out a better solution to success-oriented developers may frustrate them since their solution is no longer “successful” and requires additional work. Point out the simplicity, time-saving aspects of the new solution or how marketable the new skills required are to motivate or influence success-oriented developers. By comparison, pointing out a better solution to craftsmen may embarrass or make them feel threatened. Point to relevant code, articles or recommendations of industry thought leaders to motivate or influence craftsmen.

Also be aware the type of developer when soliciting feedback. Craftsmen generally give better feedback. Many success-oriented developers are happy to be directed by those more senior or want to direct those they perceive as junior. That said, craftsmen are likely to push newer designs and techniques while success-oriented developers look at things in the context of existing practises.

Moreover, having both developer types working together without self-awareness can create friction, such as when a success-oriented senior developer or manager asserts authority over a craftsman developer pushing what he or she considers a better solution, perhaps backed up by blogs or articles. Outsourcing or mergers/acquisitions often cause or exacerbate this.

The challenge is that developers have weak soft skills and many consider meetings and discussions unproductive time. They rarely focus on interacting with each other, sometimes oblivious to the problems it creates. Unfortunately, developers are not perfectly rational but understanding their motivations can solve common problems or avoid them completely.

(Thanks to Bryan O’Donovan for the helpful review and comments.)

Developer to Architect: A Matter of Perspective

When interviewed for a promotion some time ago, the interviewer told me two developers are arguing over whether to use programming language A or B for a project and asked me which I would recommend.

There are different ways to answer the question. The first is technical, comparing the language minutiae, libraries, IDEs and tool support. It argues the merits of the vendors, communities behind the languages and often has a bias toward favourite languages or tools.

Team leaders and managers approach the problem differently. They look at the existing skills of the team. They look at support structures (such as contracts and consultants), the longer term viability of languages and their organization’s investments in each. They consider the risks and benefits.

I gave both these answers. While correct, these answers miss a critical viewpoint: customer benefit. For example, the product may have to integrate with existing applications using mechanisms only supported by or heavily geared toward certain languages, like COM with C++, Visual Basic or C#. The target device or operating system may support one primary language, such as iOS and Objective-C. Languages may have runtimes that require additional configuration and patching, such as Java or Flash, and the customer lacks the infrastructure and expertise.

Customer benefit is an assumption so implicit in the question that people consider it assumed or ignore it. However, this assumption is what makes the question a great (trick) question, particularly for a software architect that needs to interpret, challenge and identify requirements.

“We do not see things as they are. We see them as we are.” Talmud

Moving from a developer to an architect involves a change in perspective. Developers leap to details with the “how”, arguing libraries or techniques. Managers identify resource requirements and risk. Architects step back and ask “Is this even the right question?”

“Nothing is more dangerous than an idea if it’s the only one you have.” Emil-Auguste Chartier

Thinking inside the box is easy. Thinking outside the box is hard. Finding the box is harder still. Architects may specialize but are expected to delve into enough detail to validate a proposed solution without risking leaping to conclusions or becoming too attached to a solution. Developers focus on the best and fastest way to solve a given problem.

“The limits of my language is the limit of my world” Ludwig Wittgenstein.

Design patterns are useful for conceptualizing and communicating designs. Understanding the business problem and customer viewpoint is critical to identifying the best solutions. Architects straddle both the developer and customer worlds and so must learn the terminology of both and translate concepts between them.

“It takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that!” Lewis Carroll

It is an understatement to say technology changes rapidly. Whereas developers must keep up with their chosen tools and languages in detail, architects evaluate and adapt to a wider set of technologies but less deeply than a developer, particularly those architects responsible for a technical strategy or vision.

“It is a painful thing to look at your own trouble and know that you yourself and no one else has made it.” Sophocles

Architects are responsible for the product without being the ones implementing it, including design delegated to others and trade-offs  made to handle  technical or business constraints, and their role is as much communication and evangelism as high level design. Managers have long been accustomed to this level of responsibility and facilitation but it may be new to a developer.

Of course, this is not to say a developer cannot think like an architect or manager (or other permutations) or that these are immutable, defined roles. Like Edward de Bono’s Six Hats, recognizing that there are different viewpoints and switching between them is key. It is also challenging, particularly when under the pressures of commercial software development. Or an interview.

A Glimpse at Stack Overflow

As any software developer that has searched for anything on the Internet probably knows, the Stack Exchange web sites contain an increasingly large body of useful knowledge. This initially included sites like stackoverflow.com for programming issues but has expanded into diverse areas like cooking, poker and lego. There is even a Stack Exchange site for questions about Stack Exchange.

This popularity is largely fuelled by gamification, where game design and theory is applied to ordinary tasks to make them more engaging. Stack Exchange does this primarily through reputation, similar to the XBox’s Gamerscore. Participants gain reputation primarily for asking or answering popular (up voted) questions. As participants gain reputation, they gain access to more privileges, eventually being able to effectively moderate the site.

There are also badges, akin to Xbox achievements. A bronze badge is given for simple things like completing your user profile or answering your first question. Silver and gold badges are awarded for harder things like asking or answering questions with many up votes.

Interestingly, Stack Overflow has reportedly been used by some to demonstrate competency when applying for jobs and Stack Overflow now has a careers site that requires a minimum reputation to enter. This made me wonder how difficult it would be to gain Stack Overflow reputation, say 1000, by genuinely trying to help others and participate in the intended spirit.

I registered on stackoverflow.com at 2:39pm on Thursday 30th August 2012 (Sydney, Australia time) and exceeded 1000 reputation at 10:44am on Wednesday 5th September with 850 reputation (tracked separately) on programmers.stackexchange.com, a site dedicated to software engineering and architecture rather than low-level programming issues. Indeed, Stack Exchange makes this “flair” available showing my very modest reputation and badges:

 

Although it was only six days, it was a significant time sink. Many hours were spent each day looking for questions to answer and formulating responses. Reputation (in the form of up votes) and answers often go to the first to answer so people learn quickly to provide a simple response then add more detail (and preferably sample code) later. Another strategy is seeking older, unanswered questions but the easy ones are probably already answered and such answers are unlikely to receive many up votes.

Answering questions on Stack Overflow is a lesson in humility. People often give better answers more quickly than you can type them, even in areas you consider yourself an expert, or radically different solutions. A diligently researched and prepared answer may be ignored or a negative comment may incite down votes. This can be frustrating but it can make Stack Overflow a great learning experience because the penalties for being incorrect are minor compared to the rewards of success. It is worth having a guess and let the more knowledgeable correct any mistakes.

Indeed, the reputation system provides surprisingly few opportunities for abuse. For example, someone could up vote a question they have answered (to put it back near the top of the list, looking for other’s up votes) and down vote others’ answers but up votes and reputation from up votes are capped per day. (Please note I didn’t try it. As I said above, the intention was to participate in the intended spirit.)

However, Stack Exchange is far from perfect. It does not guarantee high quality questions. Many are answered by a simple Internet search or are poorly expressed but they usually disappear quickly as the good questions are up voted. Questions on niche technologies may also languish unanswered and, when they are, lack a critical or interested population to up vote them. Like any Internet community, it is also cliquey with long established members.

Overall, I was impressed at the depth and breadth of the community and the site’s reward system. I doubt I will have the patience and dedication to reach the 400,000+ reputation some have earned but the temptation to refresh the page to see if there’s one more question you can answer is there. Answers marked correct by the question author are usually (but not always) thorough and correct and other answers or comments often contain good supporting information.

Many years ago, a potential employer could only verify one’s expertise by contacting old employers or limited technical questioning. Although not a substitute for these, Stack Exchange is one opportunity to demonstrate ability outside previous employment or education. The most telling aspect is the career’s page lets you link your best answers and highlights technologies you have participated in rather than list reputation and badges. My modest stack overflow reputation will not be appearing on my CV but maybe I’ll try to get to the next level, err, get more reputation on the stack overflow MMORPG.

Of Patents and Putters

The recent victory of Apple over Samsung has brought the arguments for and against software patents to the forefront once again. With many of the big technology companies engaged in patent litigation, many ask whether patent law is a distraction that stifles competition. Many technical luminaries, blogs and articles are openly anti-patent so should software developers be actively filing patents in organizations willing do so?

Patents can be beneficial for the developer. Some companies offer financial rewards for filing patents and researching whether something is truly novel, required for filing a patent, may reveal other solutions and techniques. However, patents’ biggest benefit is demonstrable innovation. A filed patent has been reviewed by the organization’s internal process and the patent office. This is rarely perfect but far more review than many other achievements.

That said, filing patents can distract developers from working on products and can be expensive for the organization. Similarly, if a developer files patents on a product then moves to a different organization working on similar products, the developer must carefully avoid infringing those patents at the new organization.

Some, such as the Electronic Frontier Foundation (https://www.eff.org/patent), argue that filing software patents endorses or exacerbates the negatives of patents. Many prefer the open source movement ideals where information and software should be available to be extended and used by many. Much of modern computing builds on concepts and algorithms that, if originally patented, may have stalled or prevented much of what IT takes for granted.

However, is this being disingenuous? Could these concepts and algorithms have developed despite patents? Consider the case of VisiCalc, the first spreadsheet. It was not patented, allowing Lotus 1-2-3, Microsoft Excel and other competitors to copy, extend and eventually surpass it. If it had been patented, would Lotus, Microsoft and others created an even better solution to the original problem than a spreadsheet? (as per Russ Krajec’s blog post at http://www.krajec.com/blog/what-if-visicalc-was-patented)

There are many more arguments for or against patents. However, even if all the anti-software patent arguments were proved and pro-software patent arguments disproved, for example, there is still no guarantee the software patent system would be dismantled immediately or at all. Legal and political systems move slowly at best.

Indeed, the technical or ideological pro- and anti-software patent arguments are all academic. Organizations file patents to protect their intellectual property (IP) in a world where patent litigation is increasingly common and IP is becoming organizations’ greatest asset.  It is a strategic business decision.

If an organization chooses to file patents, a software developer refusing to file patents makes about as much sense as a golfer refusing to use a putter. The golfer may lobby for a rule change banning putters but, in the meantime, the golfer is only inflating his or her handicap. This is not a “two wrongs make a right” scenario. If the organization is successful, it will inspire copy cats and litigation from its competition. Depending on the industry and product, filing patents may be pragmatic.

Software developers often see themselves only as potential victims of software patents, the expense and difficulty of filing them and the misplaced assertion that developers do not producing anything novel fuelling circular logic. If the organization is willing to bear the costs of patent management, why not patent it? There are undoubtedly smarter people out there that can solve the problem more effectively if needed. The organization can always choose not to assert the patent over open source or use it defensively, too.

Some argue that nothing in software development is new and most work is merely reinvention but saying that modern software developers are all less capable, intelligent and imaginative than their forebears is clearly false. This is not to say that developers produce new things all the time. Similarly, while many patents are obvious in retrospect, it is also much easier to judge when their ramifications are in everyday use.

What developers can do is ensure filed patents cover only products the organization intends to produce. Much of the consternation around patents from developers revolves around “non-practicing entities” (NPEs or “trolls” as they are derogatorily known), where the focus is on licensing existing patents and not using them to produce products. Research NPEs are the exception, despite Ars Technica’s Fox News-worthy rant (http://arstechnica.com/tech-policy/2012/04/how-the-aussie-government-invented-wifi-and-sued-its-way-to-430-million/).

Irrespective of whether an organization chooses to file patents, developers should keep easily accessible evidence of design documents and design discussions, such as E-mails and written notes. Even if an organization chooses not to file patents or the organization is not building anything novel, the organization may still be the victim of a patent lawsuit and demonstrable prior art can refute or diminish a patent’s novelty.

Software developers are very good at arguing things on technical merits. Unfortunately, the software patent issue has never been a technical or even ideological issue. These are merely rationalizations. This is as economic and political issue, and most software developers are not economists or politicians. Some protection for inventions is required and patents are the current, imperfect implementation of this. Until we come up a better system, software developers are only hurting themselves by not using all the rules of the game to their advantage.

In Defense of Documentation

Anyone who has spent a modicum of time working with software developers learns that developers hate creating designs, use cases, “readmes”, code samples and similar technical documentation. Agile proponents point to the “working software over comprehensive documentation” principle and Lean proponents claim documentation is “waste” since it does not directly contribute to the finished product or customer value.

They have a point. Writing clear, understandable documents is hard and it takes time away from coding. Keeping them up-to-date when inevitable changes occur is similarly time-consuming, particularly large changes or redesigns. Consequently, many stop asking for documentation, exacerbating poor or absent documentation. Many developers argue “code is documentation”. It is always up to date and developers had to work to understand it. Why shouldn’t everyone else?

However, saying “code is the only documentation” is a fallacy. Well written and designed code and indicative automated tests can be good documentation but every project includes code the developers are not proud of. Software developers often think of themselves coding like athletes lining up for a race at the Olympics. They have spent years training and are lining up to compete under perfect conditions. However, frequently code is written when developers are tired, stressed or unfamiliar with the problem or tools and this produces less correct and readable code.

Also, and arguably more important, developers are not the only audience for technical documentation. QA (testers), localization, management and documentation writers (those that write the manuals and non-technical documentation) all need technical information about the product and usually cannot read code, at least not well enough to get the information they need time efficiently. Developers new to the product also need a starting point.

Part of the problem is that documentation, particularly in larger organizations, consists of word processed documents or E-mail. E-mail in particular is a poor choice since it is only available to the recipients and often gets lost. An internal Wiki is a better choice, since most maintain a change history, have centralized storage making the latest version easier to find, support access control, provide auditing and can notify users of changes. Wikis are not perfect, however, since their storage formats are often proprietary and they often lack features of modern word pressing tools.

Wikis also encourage splitting documentation from large, monolithic documents into smaller sections. Care must be taken to ensure consistency and information is easy to find but most Wikis support free text searches. This fits better into the Agile and Lean methodologies, where smaller documents are delivered when needed rather than complete documentation delivered up front.

Writing documentation forces developers to identify and empathize with their customers or other developers, encouraging them to simplify the product’s design, user interface, installation and/or configuration. Some techniques, such as Documentation Drive Development and Readme Driven Development, encourage starting from the documentation for this reason.

It seems that literacy has become a skill in the software development industry. We have fooled ourselves into thinking that developers are only productive if they are writing code and poor documentation is acceptable. As Albert Einstein said “If you can’t explain it simply, you don’t understand it well enough.” Writing documentation forces developers out of their warm, cosy technical silos and back into the user world, reaquainting themselves with the problems the software is trying to solve.

An Architect’s Place in Agile

Scrum, the most common implementation of the Agile development methodology, has many well-defined roles. Those that contribute directly to the sprint (a unit of work usually lasting 2-4 weeks) are called “pigs”. Those that consult or assist only are “chickens”, the “scrum master” coordinates the sprint and the “product owner” prioritizes work and ensures the customer needs are met.

So where does the software architect fit in? The architect is not a pig if he or she does not write production code. Is he or she a chicken? The architect needs to be driving his or her features in the sprint and be more involved than a chicken. The architect is not responsible for team organization and a customer representative is usually the product owner.

Going back to basics, why is a software architect needed? Architects are rarely needed in projects with small, co-located teams full of senior developers working on well-defined requirements or well-understood problems. They can usually design and cooperate well enough to produce the desired results. However, large, distributed teams full of junior developers working on vague requirements or complex problems need coordination and direction. This is where architects are most useful.

One way of looking at it is Scrum is a software development methodology, not a productization methodology. Software development is one part of producing a product but there are many other parts, particularly for commercially sold software, such as business case design, marketing, licensing, documentation and localization. The architect could deliver non-functional requirements and high-level designs outside sprints like the other non-development tasks.

However, the architect need not deliver a monolithic document for the high-level design. In keeping with the Agile manifesto, as well as the Lean principle of making decisions as late as possible, the architect only needs to produce enough of the design to unblock the next sprint. The architect will still need a high-level design and identify non-functional requirements initially but Agile recognizes that design is as much a process as a product. Designs for subsequent sprints can be fleshed out in parallel with the development team, minimizing design rework as the team learns more about the problem and finds better solutions.

Could a software architect use Scrum to create the high-level design, either separate to or in parallel with the development teams? This can work if the architect has easy access to the resources he or she needs, such as customers to help understand the business problems, architects from other teams to discuss integration and development managers to check resource estimates. This cannot be guaranteed, particularly with larger, distributed groups – the cases where architects are most useful. However, it will occur in practice if the architect is providing designs for the start of each sprint.

Indeed, if the product owner is remote or often unavailable, an architect fits best into Scrum as a stand-in product owner. This breaks the Scrum rules of only having one product owner. However, different time zones, large projects and multiple commitments mean a single person cannot scale, as a former colleague of mine explained.

Development management may baulk at the perceived loss of control by making an architect a product owner. However, the word “owner” in “product owner” does not mean control of the product, merely creating, prioritizing and clarifying tasks, which architects often do anyway. Architects may not be customers but are judged whether the product meets the requirements or creates business value, just like product owners. They also know the product strategy and have spent time with the customer understanding the problem so are well-suited for this role, using their judgment to determine whether to escalate each question to the product owner.

Moreover, I think the question is not “Where does the architect fit into Agile?”, it’s “How can architects leverage Agile to better perform their role?”. For example, the architect can gain more visibility into the development team’s progress and status (through the backlog and burn down charts). The architect can present the design and gain consensus at the planning meeting that starts a sprint and (hopefully) see it working in the hand-over meeting at the end of a sprint.

Most importantly, architects must be in control of their performance rather than victims of process. A lot of smart people have worked very hard on Agile and Scrum and developers new to Scrum are advised to follow it as written, at least initially, because the reasons behind its nuances are often unclear. However, no development methodology can handle every case, and software architects are one of those things that can fall into the gaps.

%d bloggers like this: