Agile has become the norm in today’s software industry mainly due to the fact that it accelerates time to market, enables timely feedback that aids in building what the customer wants. The Key aspect that makes Agile successful is the process framework that helps guide the execution flow. Lately we have seen few instances where organizations start adopting agile, with full rigour but after few cycles, we see the process effectiveness taking a hit and organizations going back to traditional way of working. Even if agile is sustained for long, we see instances of process adherence not happening as expected and Scum Buts kind of symptoms creeping in making the organizations to think whether there was enough return on investment in transforming to agile way of doing the work.
To take a metaphor, every one of us go for periodic health checks to check if our health and take corrective and preventive measures. Corrective measures are like taking medicines for newly discovered alignments and preventive measures are like taking some precautionary measures, like consuming calcium and other multi-vitamins to prevent any feature health problems. And this periodic check-up is indeed required to ensure the healthiness of the biological process of any human body.
Similarly in any industry, there needs to be some heath-check process in place to assess the process and suggest suitable preventive and corrective measures for sustainability in the long run. We can perceive “Audits” to be one of such a Health-Check tool that checks for process consistency, continuous improvement and effectiveness, there could be many other ways also. Some organizations do regular agile assessments that look at the process adherence and effectiveness. We would be taking about audits in this article.
Many organizations that comply with the ISO and CMMI standards have a quality plan, and go through a series of audits for ISO compliance. Basically ISO requires that organizations are audited on a regular basis so that their projects are effective and are in compliance to the requirements defined in the organization’s QMS (Quality Management System). The audit requires some objective evidence to actually check if the organizations are complying with the set standards and guidelines.
Corrective action is a reactive approach to ensure conformance and Preventive action is a proactive approach to ensure conformance.
- Example of a CAR [Corrective action report]:
- Team is not doing retrospectives on a regular basis.
- Resolution: The team was distributed, do they had difficulty in doing retrospectives every sprint. They have discussed with management and decided to use online collaboration tool for doing the retrospectives every sprint that is quick and easy.
- Example of a PAR [Preventive action report]:
- Make the new team members comfortable with concepts in Agile.
- Resolution: Schedule regular internal agile refresher trainings.
Here a few reasons for an organization to go for Audit.
- Compliance to quality Management System.
- Check whether the organisation is doing what it documented.
- Assess the gaps, take corrective and preventive measures.
- Understand if the organization is complying with the set standards and goals.
- For continuous improvement across all teams in the enterprise.
- Sustainability of Agile adoption.
- Process consistency across all teams in the organization.
There are generally three types of audits:
- First party audit: Internal Audit, the company audits itself
- Second party audit: Customers audit the company
- Third party audit: Independent agency/external agency audit the company
The organization conducts internal audits using their internal auditors, identifies the gaps and works on them and then it goes for external auditing where they get audited by third party agencies. There is this third type called as customer Audits where the customers come and audit the organizations.
During the audit whether it is an internal or an external audit, the selected teams are audited, observations are recorded. Anything that is not complying with the set standards are noted down as non-conformances. After the audit is done, the observations are shared with the respective teams, discussed and corrective ad preventive actions are planned and executed. At a later stage the corrective actions are checked for implementation and then formally closed. This is the general process that happens in any organization during the Audit phase.
Now let us see how different it is for Agile Scrum Audit. Agile Scrum auditing is called as Process audit that checks if organizations are really doing Agile and not just claiming to be Agile ad also measures the effectiveness of Scrum process. I think the initial focus may be whet her the expectations are clear and to check if the compliance is there or not. If the compliance levels are good enough (satisfaction to the management) then the next level of auditing is recommended for checking the effectiveness of the process. In other words does this compliance in any way helping the organization to further improve upon. However the audit needs to be tailored in few situations to suit the organization nature of the product.
Sample audit questions:
- How are quality objectives set for teams and how?
- Team can show evidences of artefacts like Definition of Done.
- How are the product planning documents archived?
- Team can show evidences of artefacts like Product Backlog/Sprint backlog.
- How do you manage your day to day work?
- Team can show evidences of artifacts like Task Board and Sprint burn down charts.
Every organization following agile will have some minimum mandatory practices that need to be followed. These are the process workflow steps that are mandated for all the teams for consistency across the organization. These practices must be clearly worked out and defined before getting a sign-off on them as a mandate. This process mandate must be in synch with the quality management system of the organization. The Audit process basically checks if these minimum mandatory processes are followed and will look for artifacts that support the underlying process. There could be exceptions where for a particular project, Scrum may not be suitable, Kanban could be used, in such cases, the minimum mandatory process for these different kind of these processes and their applicability to the projects needs be worked out and documented separately. All deviations must be properly documented and agreed upon.
Initially when an organizations start audit the first time, it could be done in phases to check the basic process first and then check the effectiveness of the process across the entire organization. You can define the Audit scope to be basic compliance for first phase and process effectiveness for second phase. This is just an example, organizations could do it differently based on their context.
During the first phase, when the organization is doing an audit initially, it could go for auditing all the teams to check if they are complying with the Scrum process or any other agile framework. Here the audit mainly stresses on practices. In this phase, the audit mainly checks if the teams are adhering to the set of minimum mandatory process laid out in the QMS. Here the stress is on the process part and hence the Scrum Master plays a vital role in this phase. In this phase all the teams can be audited or a sample of teams can be chosen. Here the audit basically checks for compliance wrt to Scrum events like the Sprint Planning, Daily Scrum, Sprint Review, Sprint Retrospective, the Scrum team’s role like the Scrum Master, Product Owner and the Development teams, etc. The audit questionnaire clearly maps the ISO audit check points with the Scrum Process.
- SCRUM: Describe how is work planned for a Sprint?
- ISO: How do you get work items and track them to completion?
- SCRUM: How are retrospectives conducted?
- ISO: How are continuous improvements planned in your team?
- SCRUM: How often is sprint Burn down chart updated and who does that?
- ISO: How do you track work items to completion?
During the second phase, the audit could focus on the effectiveness of the process, for example are the Scrum events time-boxed and meeting their purpose? This purpose of the audit is to assess if the organization is really getting benefited due to the process change, is the process helping the organization to improve its effectiveness. Here some specific metrics could be captured and the Product Owner would be actively involved to check if the investments in the process change is worth it for the organization.
Here are some sample data points that can be used in this phase to check the process effectiveness:
- How good is the predictability?
- How is the technical debt getting addressed?
- Escaped defects?
- Say-do ratio?
- Backlog Grooming, is it effective?
- Sprint Reviews – how is the feedback captured and addressed?
- How backlogs are prepared?
- Is backlog grooming happening meticulously?
- How often is the Definition of Done and Definition of ready revisited?
- Is the code-review happening during every sprint?
- Look at the Automation test results, look at the patterns.
- How are the retrospective action items being tracked and implemented?
In this phase, the Product Owners could be audited closely to check their interaction with the teams to gather some data points around the process effectiveness. It is a good practice to review the previous audit findings at the start of the second phase audit. One can envision the first phase as to check if the compliance is in place and second place to check if the compliance is really giving some value to the organization.
As an internal auditor, I have seen an immense improvement in the flow of an organization after induction of these audit processes as many inherent issues were brought into light and addressed across all teams. Remember Audits are a means of achieving the process improvements and not a means to check only the compliance part! Again Audits are just one of the mechanisms that help to have a continual improvement, there could be other ways like agile assessments and surveys that aid this kaizen mind-set.
Originally Posted by Madhavi Ledalla @ Scrum Alliance in May 29, 2017