“Working software over comprehensive documentation” is the second value stated in the Agile Manifesto. The Manifesto’s signatories were fighting against the business and technical requirements documents synonymous with waterfall software development projects. They recognized these documents were a symptom of the failures and frustrations in many software development projects because they took too long to write, were too complex to consume, and were often outdated by the time development teams reviewed them.
The success and popularity of agile methodologies partially stem from how stakeholders and development teams collaborate. Instead of stakeholders writing lengthy requirement documents—often without a development team’s input—collaboration goes into continuous planning, user story writing, story estimating, sprint commitment, and other agile and scrum practices.
Unfortunately, some people have taken this principle too far, suggesting that agile teams don’t need documentation or don’t have to create documentation.
There’s some rationale behind these sentiments, especially when stakeholders and product owners don’t trust their development teams and request detailed documentation as a substitute. Furthermore, during the first decade of agile’s growth, creating, maintaining, searching, and using knowledgebases was a subpar experience compared to using today’s tools that integrate with the technologists’ development platforms and workflow.
Why create and maintain documentation?
Knowledge management and technical documentation maintenance are critical today because businesses are developing more software than ever before while devops teams are releasing changes frequently. The expectation is that devops teams create APIs, integrate cloud-native applications with other systems, and enhance capabilities over time. If there are incidents, security issues, or other defects, stakeholders expect agile teams to fix and release changes easily.
No one wants a new decade of legacy systems and technical debt that development teams will struggle to maintain and improve. Expecting new developers to join software development projects and learn how a system functions without documentation is unrealistic.
Furthermore, data scientists often use the underlying data created by microservices and applications to generate analytics, data visualizations, and machine learning models. Data scientists also seek documentation on data definitions and business rules in order to use data correctly and understand how applications create and update data.
Lastly, many organizations operate in regulated industries where user interfaces, workflows, algorithms, business logic, data catalogs, and system changes require supporting documentation.
Identify the primary audiences and use cases
It’s important to recognize that software, data, and architecture documentation have several consumers and use cases:
- Development teams actively working on microservices and applications who need to dive into aspects of systems created by other teams they collaborate with on dependencies and releases
- IT operations, business operations, infosec, IT service management specialists, and site reliability engineers who want to understand architecture, workflows, business logic, data, and error conditions
- Product owners, user experience specialists, designers, and marketers who want to understand journey maps, web analytics, and end-user usage patterns
- Data governance specialists who want to review data for security, privacy, and compliance considerations
- Data scientists, dataops engineers, and data stewards who want to ensure data quality, establish master data sources, and leverage application data sources in downstream analytics and machine learning models
- New technologists joining the organization, including developers, business analysts, test automation engineers, and devops engineers tasked to maintain and enhance code developed by their predecessors
Following naming conventions, creating readable code, and documenting the code are useful to the current and future devops teams, but they’re not sufficient for supporting many other organization functions and use cases. Code documentation and requirements embedded in user stories are also insufficient to support new devops team members who likely need top-down documentation on the architecture, data, APIs, business logic, and user interfaces before code-level documentation becomes useful.
A best practice is to identify the stakeholders of the documentation and prioritize their primary needs. Agile teams should consider itemizing documentation needs in their backlogs and agree on conventions. It’s appropriate to define acceptance criteria if documentation is expected with a user story, such as creating API documentation when developers code them. It may also be beneficial to establish a custom Jira issue type, Azure DevOps work item type, or another type of card on the scrum backlog to signify a documentation deliverable.
Standardize knowledge management and documentation tools
Since code documentation only serves the devops teams, IT leadership should define the knowledge management tools, information architecture, and documentation standards to address all stakeholder needs.
From my days as a startup CTO and enterprise CIO, I have some strong opinions on tools and standards. Organizations may use Microsoft 365 or Google Workspace to create and manage documents, presentations, and spreadsheets, but they aren’t optimal knowledgebase platforms for agile development and devops teams. A useful engineering document is often a mix of diagrams, tables, and highly formatted documented sections. Typical office applications aren’t the best option.
Furthermore, the connection between documents can often be more useful than the documents themselves. For example, a data scientist may start with a diagram, click into a service, review an API, then connect to a data dictionary to learn how to leverage an application data source correctly.
I recommend that IT and devops teams seek knowledge management tools that connect to their workflows, enable a rich information architecture, and create consumable documentation for all stakeholders.
Atlassian Confluence is a knowledge management tool recently featured as InfoWorld 2021 Technology of the Year winner. More than 60,000 customers use Confluence’s enterprise wiki for documentation, knowledge sharing, and collaboration with Confluence templates and integrations. Development teams that use Jira Software can create custom views of their backlogs using Jira Macros and incorporate diagrams into wiki pages with tools such as Gliffy, Miro, Balsamiq, draw.io, or Lucidchart.
Other documentation and knowledge-sharing tools popular with devops teams include Microsoft Teams wiki and file sharing, Microsoft SharePoint, BMC’s ComAround, ServiceNow Knowledge Management, ProProfs Knowledge Base, and other knowledge management tools. Some organizations also review document collaboration tools and portals focused on knowledge sharing, such as Bit, Igloo, Jive, LumApps, and Simpplr.
Development organizations should also consider tools for documenting specialized technology functions. For example, devops can use Swagger for API documentation, Resolve Insight’s Discovery and Dependency Mapping to capture hybrid cloud infrastructure configurations, and cloud architecture diagramming tools. Organizations establishing data science, self-service data visualization, data governance, dataops, machine learning, and other analytics capabilities should also centralize data catalogs.
Establish documentation standards and workflow
Once organizations have identified audiences, prioritized a documentation backlog, and selected tools, they should consider creating documentation standards and templates. Devops teams must keep end-users in mind when creating documentation. Centralizing the content without documentation and information architecture standards can make it less useful.
If you’re applying agile practices to creating and maintaining documentation, ask the end-users to help define acceptance criteria for the documentation they request and product owners prioritize. When documents are delivered, capture a customer satisfaction score and use it to help guide documentation standards. The documents with higher scores can become templates for future documentation needs.
The other standard to define is when and who needs to create, update, and maintain documentation. With devops teams seeking to release frequently, leaving behind obsolete documentation or documentation debt is irresponsible. A best practice is to define documentation standards and expectations as part of release and change management processes.
Organizations committed to sustainable agile development, devops, and data science must create knowledge-sharing governance and practices. Technology changes too quickly and technologists want to move on to new assignments. Creating a culture that champions documentation and knowledge sharing is critical for long-term success.