Importance of Metrics for Agile Scrum Environments: Comparison
View Latest Version
Please note this is a comparison between Version 2 by Alfred Zheng and Version 1 by Fernando Almeida.

Metrics are key elements that can give us valuable information about the effectiveness of agile software development processes, particularly considering the Scrum environment. Scrum was conceived by Jeff Sutherland and Ken Schwaber in 1993 with the intention of being a faster, more effective, and more reliable way to develop software for the technology industry.

  • agile
  • Scrum
  • metric
  • software development
  • software engineering
  • project management

1. Introduction

Agile methodologies have gained great importance in the project management field, having been strongly influenced by Japanese philosophy. As argued by Poth et al. [1], the practices related to planning, controlling, and streamlining are actions strongly related to techniques and principles of Lean production that can be applied to any industry with the goals of reducing waste and creating value. Within the agile methodologies, people can find different methods such as Kanban, Lean, Scrum, Extreme Programming (XP), and the Rational Unified Process (RUP), among others. Data obtained by KPMG in 2019 [2] indicate that 91% of organizations consider the adoption of agile in their organizations a priority and Digital.ai [3] registered an increase from 37% in 2020 to 86% in 2021 in the number of agile adoptions in software teams, with Scrum standing out as the most popular framework, followed by Kanban and Lean.
The agile manifesto emerged in 2001 when a group of 17 representatives from various software development practices and methodologies met to discuss the need for lighter and faster alternatives to the existing traditional methodologies. From this meeting, the Agile Alliance presented the Manifesto for Agile Software Development to elucidate the approach known today as agile development. The values of the agile methodology are based on four pillars [4]: (i) individuals and interactions over processes and tools; (ii) working software over comprehensive documentation; (iii) customer collaboration over contract negotiation; and (iv) responding to change over following a plan. Agile methods were designed to use a minimum of documentation, helping in the flexibility and responsiveness to change, that is, in this methodology, flexibility and adaptability are much more important than planning, unlike the traditional methodology [5,6,7,8,9,10][5][6][7][8][9][10].
Scrum was conceived by Jeff Sutherland and Ken Schwaber in 1993 with the intention of being a faster, more effective, and more reliable way to develop software for the technology industry [11]. This method emerged motivated by the traditional method, called waterfall, being too slow and often resulting in a product not desired by the customer and more expensive [12,13][12][13]. Alternatively, agile methodologies present an incremental and iterative process whose objective is to identify the priority tasks in each phase and effectively manage time with efficient teams [14,15,16,17,18][14][15][16][17][18]. Therefore, agile methodologies came to face the difficulties that occurred during project management.
The processes inherent to the several phases of process management must be effective and efficient. According to Flores-Garcia et al. [19], the advances in technology that have occurred in the last decades have provided business managers with a large volume of tools to help them make decisions. This new business scenario has forced development companies to constantly seek technologies and methods that allow them to guarantee the quality of the products offered so that they do not lose competitiveness in the market where they operate. In this context, the use of metrics that can define the performance of projects, as well as the products resulting from them, has taken on an increasingly important role in the industry and, consequently, in academia, which is preparing to meet the challenges posed by these organizations.
The most well-known and commercially successful software companies such as Google or IBM have adopted metrics to evaluate the success of their project management and campaigns [20,21][20][21]. Indeed, it is not only in the waterfall development model that it is necessary to have metrics to evaluate the performance of your processes and teams. In addition, in agile development, it is necessary to measure the effectiveness and efficiency of the processes using metrics. Planning and monitoring are necessary for projects developed in Scrum [22]. Previous works developed by Almeida & Carneiro [23], Kurnia et al. [24], and López et al. [25] were important in synthesizing these metrics considering the whole Scrum development cycle and its ceremonies. However, none of these studies provide a measure of the relative importance of these metrics considering the various agile Scrum roles (i.e., Product Owner, Scrum Master, and development team). Understanding the importance of these metrics while considering the specific role of each Scrum role is important to increase team cohesion and the quality of the work produced. It is also a way to gain practical insight into the role of each Scrum role in teams and establish policies to promote increased effectiveness and efficiency of project management in Scrum.

2. Perceived Importance of Metrics for Agile Scrum Environments

Velimirovic et al. [26] note that to monitor the progress of projects and promote the necessary improvements during their execution, the use of performance indicators is necessary. Therefore, before starting project management, the first step to ensure success is to define the metrics for follow-up. Having performance indicators for each stage of project implementation is essential to optimize the results and guide the team’s path. In the software industry, metrics are used for several reasons, such as project planning and estimation, project management and monitoring, understanding quality and business goals, and improving communication, processes, and software development tools [27]. The first paper identifying metrics for the Scrum environment was developed in 2015 by Kupiainen et al. [28]. This study sought to provide identifications collected from scientific papers and metrics used in the industry. Despite this dual aspect, the methodology used did not involve the collection of primary metrics but only a survey of secondary sources through the development of a systematic literature review. The rationale for and effects of the use of metrics in areas such as sprint planning, software quality measurement, impediment resolution, and team management were identified. The results of this study allowed us to conclude that the most important metrics are related to customer satisfaction and backlog progress status monitoring, considering the product specifications and the work developed in each sprint. Other studies have been undertaken. Kurnia et al. [24] collected 34 metrics related to Scrum development. The metrics included the entire Scrum development cycle, such as sprint planning, daily meetings, and retrospective sessions. The findings contributed to the identification of the most common metrics in the literature and identified new metrics related to the value delivered to the customer and the rate of development throughout the sprint. In Almeida & Carneiro [23], a review of Scrum metrics was performed considering a primarily quantitative study with software engineers. The study involves 137 Scrum engineers and concluded that “delivered business value” and “sprint goal success” were the most relevant metrics. In López et al. [25], a total of 61 studies from the last two decades were reviewed to explore how quality is measured in Scrum. Two important conclusions were drawn from this study. First, despite a large body of knowledge and standards, there is no consensus regarding the measurement of software quality. Another conclusion is that there is a very diverse set of metrics for this purpose, with the top three being related to performance, reliability, and maintainability. In Kayes et al. [29], a different perspective was used, with the goal of proposing a metric for measuring the quality of the testing process in Scrum. Therefore, instead of looking at the whole Scrum cycle, attention was focused on a specific action. The Product Backlog Rating (PBR) provides a complete picture of the testing procedure of a product throughout its development cycle. The PBR takes into consideration the complexity of the features to be produced in a sprint, evaluates the test ratings, and provides a numerical score of the testing process. A similar line of research is the work conducted by Erdogan et al. [30], which looked at the value of metrics in the process of analyzing sprint retrospectives. This study found metrics for the inspection process, improving team estimates and increasing team productivity. This study further found that the metrics of “actual quality activity effort rate” and “subcomponent defect density” helped improve product quality. The metrics collected as part of this study allowed us to synthesize the metrics and their definition, as shown in Table 1. Metrics related to effort estimation can be used to prioritize features to be developed or to prioritize activities based on relative value and effort, and velocity can be used to improve effort estimates for the next iteration, which will help the team to verify that the planned scale has been completed. Metrics related to defect identification can be used to inspect the defects in the backlog, which will allow the sharing of this information among the team members. In addition, in the same vein, people found metrics that measure defects that appear during a sprint. Finally, there are metrics, such as return on investment, that measure the delivery of software and that can be used to understand the relationship between the result and the investment in software.
Table 1.
Overview of Scrum metrics.
Activity Metric Definition
Daily scrum Number of tasks Total number of tasks in the sprint backlog.
Number of tasks in progress Number of tasks “in progress” in the sprint backlog.
Number of concluded tasks Number of tasks completed in the sprint backlog.
Estimated hours for a task Time needed in hours to complete a task.
Remaining hours for a task Remaining time in hours to complete a task.
Number of impediments Number of impediments, obstacles, or issues that hinder the progress of a Scrum team during the implementation of a sprint.
  Workload distribution Measure of how much work is assigned to each development member for the current sprint.
Product backlog Number of user stories Total number of user stories in the product backlog.
Number of added user stories Number of new user stories added to the product backlog.
Number of deleted user stories Number of user stores removed from the product backlog.
Business value Importance of a user story considering the product owner’s vision. It should reflect the value generated for the organization in terms of revenue, customer satisfaction, market share, competitive advantage, or any other relevant business objective.
Sprint backlog Number of user stories Number of user stories in the sprint backlog.
Number of tasks Number of tasks in the sprint backlog.
Hours spent to implement a task Hours spent in a day to implement a given task.
Hours remaining to finish a sprint Time in hours remaining to finish the current sprint.
  Sprint burndown Graphic representation of the rate at which work is completed and how much work remains to be performed in a sprint.
Sprint planning meeting Sprint length Duration of a given sprint.
Size of team Number of developers in the development team.
Team members’ engagement Level of engagement of the team member in their work and workplace.
Sprint retrospective Number of tasks in a sprint Number of tasks assigned to a sprint.
Number of tasks completed in a sprint Number of tasks completed in a sprint.
Number of user stories completed in a sprint Number of user stories implemented during the sprint.
Sprint review Number of accepted user stories Number of user stories accepted by the customer during the sprint review.
Number of rejected user stories Number of user stories rejected by the customer during the sprint review.
Team performance Accuracy of estimation Percentage of correctness of the estimated implementation time of the user stories compared to their actual implementation.
Focus factor The speed of implementation to be divided by the internal capacity of the team.
Targeted value increase Team’s speed in the current sprint divided by its initial speed.
Team member turnover Indicates the turnover of team members considering a full development cycle.
Team satisfaction Degree of satisfaction of the team with the Scrum environment and adopted methodologies.
Velocity Amount of work a development team can do during a sprint. It can be calculated by considering the story points divided by actual hours or the estimated hours divided by actual hours.
Work capacity The total time the team is available for work during a sprint. It is usually measured in hours.
Tests Acceptance tests per user story Number of acceptance tests per user story.
Defects count per user story Total number of defects per user story
Defects density Number of defects found divided by the size of the considered module/software.
Functional tests per user story Number of functional tests per user story.
Tests automation percentage Tests automation percentage considering automatic tests and manual tests.
  Unit tests per user story Number of unit tests per user story.
Scrum governance is a challenging task because it cannot focus only on software development processes and must involve multiple domains and include interdisciplinarity members. Empirical studies developed by [31,32,33,34][31][32][33][34] show that organizations implementing Scrum must concentrate their efforts on process improvement in a controlled and limited number of areas to face the high complexity of the continuous improvement processes and the strong interconnection between them. This requires keeping track of the metrics of the Scrum activities. As reported by Kadenic et al. [35], the experience of a Scrum professional in understanding the Scrum work processes is an important element for the success of the implementation and diffusion of Scrum in the organization.

References

  1. Poth, A.; Sasabe, S.; Mas, A.; Mesquida, A.L. Lean and agile software process improvement in traditional and agile environments. J. Soft. Evol. Proc. 2019, 31, e1986.
  2. KPMG. Agile Transformation. Available online: https://assets.kpmg/content/dam/kpmg/be/pdf/2019/11/agile-transformation.pdf (accessed on 26 January 2023).
  3. Digital.ai. 16th State of Agile Report. Available online: https://digital.ai/resource-center/analyst-reports/state-of-agile-report/ (accessed on 26 January 2023).
  4. Beck, K.; Beedle, M.; van Bennekum, A.; Cockburn, A.; Cunningham, W.; Fowler, M.; Grenning, J.; Highsmith, J.; Hunt, A.; Jeffries, R.; et al. Manifesto for Agile Software Development. Available online: https://agilemanifesto.org (accessed on 26 January 2023).
  5. Dingsoyr, T.; Nerur, S.; Balijepally, V.; Moe, N.B. A decade of agile methodologies: Towards explaining agile software development. J. Syst. Soft. 2012, 85, 1213–1221.
  6. Lindskog, C.; Netz, J. Balancing between stability and change in Agile teams. Int. J. Manag. Proj. Bus. 2021, 14, 1529–1554.
  7. Koi-Akrofi, G.Y.; Koi-Akrofi, J.; Matey, H.A. Understanding the characteristics, benefits and challenges of agile IT project management: A literature based perspective. Int. J. Soft. Eng. Appl. 2019, 10, 25–44.
  8. Venkatesh, V.; Thong, J.Y.L.; Chan, F.K.Y.; Hoehle, H.; Spohrer, K. How agile software development methods reduce work exhaustion: Insights on role perceptions and organizational skills. Inf. Syst. J. 2020, 30, 733–761.
  9. Bonner, N.A.; Kulangara, N.; Nerur, S.; Teng, J.T. An Empirical Investigation of the Perceived Benefits of Agile Methodologies Using an Innovation-Theoretical model. J. Database Manag. 2016, 27, 38–63.
  10. Altameem, E. Impact of Agile Methodology on Software Development. Comp. Inf. Sci. 2015, 8, 9–14.
  11. Sutherland, J. The Art of Doing Twice the Work in Half the Time; Crown Business: New York, NY, USA, 2014.
  12. Thesing, T.; Feldmann, C.; Burchardt, M. Agile versus Waterfall Project Management: Decision Model for Selecting the Appropriate Approach to a Project. Proc. Comp. Sci. 2021, 181, 746–756.
  13. Yu, J. Research Process on Software Development Model. IOP Conf. Ser. Mater. Sci. Eng. 2018, 394, 032045.
  14. Rubin, K. Essential Scrum: A Practical Guide to the Most Popular Agile Process; Addison-Wesley: Boston, MA, USA, 2012.
  15. Behrens, A.; Ofori, M.; Noteboom, C.; Bishop, D. A systematic literature review: How agile is agile project management? Issues Inf. Syst. 2021, 22, 278–295.
  16. Madadopouya, K. An Examination and Evaluation of Agile Methodologies for Systems Development. Aust. J. Comp. Sci. 2015, 2, 1–17.
  17. Fagarasan, C.; Popa, O.; Pisla, A.; Cristea, C. Agile, waterfall and iterative approach in information technology projects. IOP Conf. Ser. Mater. Sci. Eng. 2021, 1169, 012025.
  18. Lalband, N.; Kavitha, D. Software Development Technique for the Betterment of End User Satisfaction using Agile Methodology. TEM J. 2020, 9, 992–1002.
  19. Flores-Garcia, E.; Bruch, J.; Wiktorsson, M.; Jackson, M. Decision-making approaches in process innovations: An explorative case study. J. Manuf. Technol. Manag. 2021, 32, 1–25.
  20. IBM. Use Key Performance Indicators to Measure and Guide Progress. Available online: https://www.ibm.com/garage/method/practices/learn/kpis-measure-guide-progress/ (accessed on 31 January 2023).
  21. Eriksson, J. KPIs: An Essential Framework. Available online: https://www.thinkwithgoogle.com/intl/en-145/future-of-marketing/creativity/kpis-essential-framework/ (accessed on 31 January 2023).
  22. Tekin, N.; Yilmaz, M.; Clarke, P. A novel approach for visualization, monitoring, and control techniques for Scrum metric planning using the analytic hierarchy process. J. Softw. Evol. Proc. 2021, e2420.
  23. Almeida, F.; Carneiro, P. Performance metrics in scrum software engineering companies. Int. J. Agile Syst. Manag. 2021, 14, 205–223.
  24. Kurnia, R.; Ferdiana, R.; Wibirama, S. Software Metrics Classification for Agile Scrum Process: A Literature Review. In Proceedings of the International Seminar on Research of Information Technology and Intelligent Systems (ISRITI), Yogyakarta, Indonesia, 21–22 November 2018; pp. 174–179.
  25. López, L.; Burgués, X.; Martínez-Fernández, S.; Vollmer, A.M.; Behutiye, W.; Karhapäa, P.; Franch, X.; Rodríguez, P.; Oivo, M. Quality measurement in agile and rapid software development: A systematic mapping. J. Syst. Softw. 2022, 186, 111187.
  26. Velimirovic, D.; Milan, V.; Rade, S. Role and importance of key performance indicators measurement. Serb. J. Manag. 2011, 6, 63–72.
  27. Staron, M.; Meding, W.; Niesel, K.; Abran, A. A Key Performance Indicator Quality Model and Its Industrial Evaluation. In Proceedings of the 2016 Joint Conference of the International Workshop on Software Measurement and the International Conference on Software Process and Product Measurement (IWSM-MENSURA), Berlin, Germany, 5–7 October 2016; pp. 170–179.
  28. Kupiainen, E.; Mäntylä, M.V.; Itkonen, J. Using metrics in Agile and Lean Software Development—A systematic literature review of industrial studies. Inf. Soft. Technol. 2015, 62, 143–163.
  29. Kayes, I.; Sarker, M.; Chakareski, J. Product backlog rating: A case study on measuring test quality in scrum. Innov. Syst. Soft. Eng. 2016, 12, 303–317.
  30. Erdogan, O.; Pekkaya, M.E.; Gök, H. More effective sprint retrospective with statistical analysis. J. Softw. Evol. Proc. 2018, 30, e1933.
  31. Ayunda, P.L.; Budiardio, E.K. Evaluation of Scrum Practice Maturity in Software Development of Mobile Communication Application. In Proceedings of the 2020 3rd International Conference on Computer and Informatics Engineering (IC2IE), Yogyakarta, Indonesia, 15–16 September 2020; pp. 317–322.
  32. Panjaitan, I.; Legowo, N. Measuring Maturity Level of Scrum Practices in Software development Using Scrum Maturity Model. J. Syst. Manag. Sci. 2022, 12, 561–582.
  33. Pino, F.J.; Pedreira, O.; García, F.; Luaces, M.R.; Piattini, M. Using Scrum to guide the execution of software process improvement in small organizations. J. Syst. Soft. 2010, 83, 1662–1677.
  34. Baxter, D.; Turner, N. Why Scrum works in new product development: The role of social capital in managing complexity. Prod. Plann. Contr. 2021, 179, 109459.
  35. Kadenic, M.D.; Koumaditis, K.; Junker-Jensen, L. Mastering scrum with a focus on team maturity and key components of scrum. Inf. Soft. Technol. 2023, 153, 107079.
More
© Text is available under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/)
Video Production Service
Feedback