Ontology-Based Regression Testing: Comparison
Please note this is a comparison between Version 2 by Jason Zhu and Version 1 by Muhammad Hasnain.

Regression testing is a relevant research field focused on ensuring software works correctly after being modified or when new functionalities are added. The widespread use of information technology techniques has enabled the rapid development of applications.

  • Regression Testing
  • semantic rules
  • scalability
  • surveillance systems
  • security

1. Introduction

Modifications or the addition of new features in the applications require retesting the web applications, which adds the problem or concern for software testers to retest the applications. Retesting web services/web applications is a relatively hard job for testers, and it also incurs a high cost. ‘ Regression Testing’ (RT) with its subtypes ‘test case selection’, ‘test case reduction’, and ‘test case prioritization’ have been widely examined in research studies [1]. Test case selection approaches aim at increasing the effectiveness based on their measurement capabilities, including the cost, coverage, and fault detection [2,3,4][2][3][4]. Test case prioritization (TCP) approaches let researchers’ reorder test cases to reveal maximum faults based on specific criteria. Test cases with the top defects are tested before the test cases with fewer failures [5]. Proposed RT approaches rely on some criteria, and the ontology of systems is one of these criteria proposed to perform test case selection, reduction, and prioritization.

Ontology refers to basic concepts and their relationship in a research domain [6]. Due to ontology reasoning, the ideas which are harder for their explicit expression are derived. Ontology provides further analysis and assessment knowledge, including the accurate classification of faulty and non-faulty software engineering modules. The field of ontology covers the aspects of methodologies and metrics applied to solve issues in software engineering [7]. OntoClean is one of these methods that covers the use of some properties, including unity, rigidity, identity, and dependency [8]. The OntoClean method, along with meta-properties, is proposed to provide entities’ semantic and logical meaning. In [9], researchers made extensions to define ontology by expressing vocabulary concepts and their use. In this entire document, web applications and web services are interchangeably used.

The motivation behind this ‘systematic literature review’ (SLR) is to investigate the continuous lack of discussion on ontology-derived regression testing in several SLRs [10]. Ontology-derived testing can be considered under the domain of regression testing [11].

Table 1. Existing SLRs on regression testing.
Sr. No Research Focus Ontology-Based Regression Testing Number of Scholarly Studies Reference
1.  Regression Testing x 159 [9]
2.  Regression test case prioritization x 65 [10]
3.  Test case prioritization x 120 [11]
4.  Regression testing of web services 30 [12]
5.  Regression test case selection x 47 [3]
6.  Test case prioritization x 69 [13]
7.  Test Case prioritization of systems x 90 [14]
8.  Testing of semantic web services 43 [15]
9.  Agent-based test generation x 115 [16]

Table 1 is the illustration of SLRs in the area of regression testing. The first SLR by Qui et al. [12] includes two papers discussing web services’ semantic behavior. Studies [13,14][13][14] did not include papers on ontology-based regression testing. A study by de Souza et al. [15] presented a description of web services ontologies. They include two ontology-driven models, ‘web services modeling ontology’ (WSMO) and ‘web services description language semantics’ (WSDL-S). Therefore, our SLR’s motivation is to help researchers and software practitioners working on the ontologies in regression testing by presenting the state-of-the-art ontology-based RT approaches.

2. Research Method

Below are the research questions studied for this SLR (see Table 2 ).

Table 2. Proposed research questions.
RQ ID. Description
RQ1 What is the roadmap of regression testing?
RQ2 What are the state-of-the-art techniques of ontology-based regression testing?
RQ3 What are the challenges and limitations of current approaches?
RQ4 What are the possible future research directions?
RQ5 What are the unique issues of ontology-based regression testing compared to other regression testing approaches?

A set of keywords was constructed using the words and phrases from research questions. Search strings relevant to the topic of interest and research questions were formulated [28][17]. Researchers used well-known research repositories, including IEEE Xplore, ACM digital library, ScienceDirect, Springer Link, Web of Science, and Wiley & Sons, to search the research articles published between 2007 and 2019. The search strings applied to the digital repositories mentioned above are shown in the following Table 3 .

Table 3. Search strings used for paper’s retrieval.
Data Repository Name Search String
IEEE Xplore (((((“All Metadata”: Ontology) AND “All Metadata”: Regression Testing) OR “All Metadata”: Test Case Prioritization) AND “All Metadata”: Test Case Selection) AND “All Metadata”: Test Case Generation)
ACM digital Library (+Ontology + Regression + Testing + Test + Case + Prioritization + Test + Case + Selection + Test + Case + Generation)
ScienceDirect “Ontology, regression testing, test case prioritization, Test Case Selection, Test Case Generation.”
SpringerLink ‘Ontology AND “Regression Testing” AND (Test OR Case OR Prioritization, OR Test OR Case OR Selection, OR Test OR Case OR Generation)’
Web of Science (Ontology) AND TOPIC: (Regression Testing) OR TOPIC: (test case prioritization) OR TOPIC: (Test Case Selection) OR TOPIC: (Test Case Generation)
Wiley & Sons “Ontology” anywhere and “Regression Testing” anywhere and “Test Case Prioritization” anywhere and "Test Case Selection” anywhere and “Test Case Generation”.

It was found that when wresearchers applied “regression testing” OR “test case prioritization” strings, many irrelevant studies to the topic of this SLR were returned. WeResearchers refined ourthe search strings using Boolean ‘AND’ and ‘OR’ operators between search keywords to find relevant publications.

The obtained research documents were examined by two independent reviewers (MH and IG). This phase mainly aimed at verifying the research articles, which were potentially eligible and could be used for further analysis. At the screening phase, research documents were screened via title, abstract, and keywords. However, two reviewers (MFP and SRJ) read and reviewed full-length documents resulting from the screening phase at the eligibility phase. Since ontology has been used in software testing, reviewers included papers related to the topic “ontology-based regression testing” with great caution in this systematic review. Therefore, 24 research documents were included in this study.

3. RCurresults and Dint Studiescussion

In the following Table 54 , wresearchers summarize the studies that focused on ontologies in proposing approaches in regression testing.

Table 54. Summary of ontology-based approaches.
Authors Problem Proposed Technique Advantage Limitation
Askarunisa et al. [37]Askarunisa et al. [18] High cost on web services testing An automated testing framework Effective prioritization of web services No cost resources were used in the proposed approach
Askarunisa et al. [38]Askarunisa et al. [19] An increase in cost due to source code unavailability Semantic-based protégé tool Faults detection is better than the traditional approaches Coverage criteria is not clearly mentioned
Bai et al. [40]Bai et al. [20] Web services testing in an open platform A risk-based approach using semantics Faults detection with a high impact The reliability of web services is not mentioned
Kim et al. [41]Kim et al. [21] Handling changes in the evolving system is a difficult task Regression test selection Reduces the overall time to rerun test cases This approach might have missed some necessary test cases
Fan et al. [43]Fan et al. [22] Validation of tests in an ad hoc fashion Domain-specific ontology-based approach Systematic generation of TCG from safety analysis report The cost to ensure the safety of a web system is not mentioned
Tseng et al. [44]Tseng et al. [23] Uncertain test coverage from random testing Domain-specific ontology using requirements Ensures users’ safety, more effective than random testing Validation of web systems at large-scale systems is not done

Table 54 shows ontology-based studies and authors’ information, problems, proposed techniques, advantages, and limitations of the proposed approaches. The proposed approaches mainly addressed cost, testing in an open platform, modification in evolving web systems, and validation of test cases in nuclear safety systems. This SLR shows the remarkable penetration of ontologies in the prioritization of test cases in web services. Ontology-based regression testing approaches were found effective in saving costs and time compared with random testing approaches. Semantics-based designing of approaches can be further used with the help of ‘artificial intelligence’ to monitor the continuous performance of web services.

A study [40][20] proposed a risk assessment approach for the ranking of test cases. Researchers in this study considered risk assessment in the context of users’ input/output and pre/post conditions, processes, and ontology constraints. The risk assessment covers the dissatisfaction of regulatory/legal requirements and relates mainly to a process, work, or project that does not achieve the expected functional behavior. The proposed TCP approach does not involve the reliability of web services. A product risk carefully refers to security and safety behaviors [48][24], as well as it also concerns the reliability behavior of the web services. Therefore, web services’ reliability factor can improve regression testing and prioritize test cases. In addition to the challenges mentioned earlier, the failure of critical systems is closely related to the proposed maintenance approach [49][25], which cannot detect earlier faults. It is still challenging for researchers to optimize the test case selection and TCP in regression testing.

Before SLR, de Souza et al. [15] identified the research gap that the model-based testing has not dealt with because abstract test cases are trivial to map them to executable and concrete test cases. Compared with the paper [22][26], scalability is a focused research problem regarding test brokers because they face re-execution of test cases of a suitable size. The scalability issue remains a commonly identified challenge for both ontology-based RT and collaborative testing of web services. Due to web services’ evolution, scalability is still an open issue in the software testing domain. Validation of proposed RT approaches is mentioned as a problem [50][27] because all proposed simulation-based approaches do not address the testing of web applications in real-world scenarios. Therefore, the validation problem is close to our identified problem. A study [19][28] stated that cloud testing faces the scalability challenge because distributed systems need enhancement in their performance over time. This performance enhancement can combat the increasing workload on web services. Finally, inadequate validation of test cases is a unique challenge for automated test case generation [51][29]. Validation of the test cases is crucial for real-time processing to reduce software testing costs and effort.

4. Conclusions and Future Implications

This SLR identified ontology-based RT approaches proposed between 2007 and 2019. Most of the studies focused on TCG and TCP research topics. Our SLR presented the roadmap by identifying the benchmark approaches along with their objectives. This study identified six challenges of ontology-based RT approaches. The cost and validation were difficulties widely examined in the selected studies. Scalability, risk assessment, failure of critical systems, and reliability are still open regression testing issues. Our SLR presented the effectiveness of ontology-based RT approaches for addressing the failure in systems other than bioinformatics. This SLR’s findings suggested developing more specific simulation tools regarding the security of the critical system in future works.

References

  1. Chen, J.; Zhu, L.; Chen, T.Y.; Towey, D.; Kuo, F.C.; Huang, R.; Guo, Y. Test case prioritization for object-oriented software: An adaptive random sequence approach based on clustering. J. Syst. Softw. 2018, 135, 107–125.
  2. Spieker, H.; Gotlieb, A.; Marijan, D.; Mossige, M. Reinforcement learning for automatic test case prioritization and selection in continuous integration. In Proceedings of the ISSTA 2017: 26th ACM SIGSOFT International Symposium on Software Testing and Analysis, Santa Barbara, CA, USA, 10–14 July 2017; pp. 12–22.
  3. Kazmi, R.; Jawawi, D.N.; Mohamad, R.; Ghani, I. Effective Regression Test Case Selection: A Systematic Literature Review. ACM Comput. Surv. 2017, 50, 1–32.
  4. de Oliveira Neto, F.G.; Torkar, R.; Machado, P.D. Full modification coverage through automatic similarity-based test case selection. Inf. Softw. Technol. 2016, 80, 124–137.
  5. Krishnamoorthi, R.; Mary, S.S.A. Factor oriented requirement coverage based system test case prioritization of new and regression test cases. Inf. Softw. Technol. 2009, 51, 799–808.
  6. Koutsomitropoulos, D.A.; Kalou, A.K. A standards-based ontology and support for Big Data Analytics in the insurance industry. ICT Express 2017, 3, 57–61.
  7. García-Peñalvo, F.J.; Colomo-Palacios, R.; García, J.; Theron, R. Towards an ontology modeling tool. A validation in software engineering scenarios. Expert Syst. Appl. 2012, 39, 11468–11478.
  8. García, J.; Jose’García-Penalvo, F.; Therón, R. A Survey on Ontology Metrics. In World Summit on Knowledge Society; Springer: Berlin/Heidelberg, Germany, 2010; pp. 22–27.
  9. Yoo, S.; Harman, M. Regression testing minimization, selection and prioritization: A survey. Softw. Test. Verif. Reliab. 2010, 22, 67–120.
  10. Singh, Y.; Kaur, A.; Suri, B.; Singhal, S. Systematic literature review on regression test prioritization techniques. Informatica 2012, 36, 308–379.
  11. Catal, C.; Mishra, D. Test case prioritization: A systematic mapping study. Softw. Qual. J. 2013, 21, 445–478.
  12. Qiu, D.; Li, B.; Ji, S.; Leung, H.K.N. Regression Testing of Web Service: A Systematic Mapping Study. ACM Comput. Surv. 2015, 47, 1–46.
  13. Khatibsyarbini, M.; Isa, M.A.; Jawawi, D.N.; Tumeng, R. Test case prioritization approaches in regression testing: A systematic literature review. Inf. Softw. Technol. 2018, 93, 74–93.
  14. Mukherjee, R.; Patnaik, K.S. A survey on different approaches for software test case prioritization. J. King Saud Univ. Comput. Inf. Sci. 2018, 33, 1041–1054.
  15. de Souza Neto, J.B.; Moreira, A.M.; Musicante, M.A. Semantic Web Services testing: A Systematic Mapping study. Comput. Sci. Rev. 2018, 28, 140–156.
  16. Arora, P.K.; Bhatia, R. A Systematic Review of Agent-Based Test Case Generation for Regression Testing. Arab. J. Sci. Eng. 2017, 43, 447–470.
  17. Kitchenham, B.A.; Brereton, O.P.; Budgen, D. Protocol for Extending an Existing Tertiary Study of Systematic Literature Reviews in Software Engineering; EPIC Technical Report. EBSE-2008-006 June 2008; University of Durham: Durham, UK, 2017.
  18. Askarunisa, A.; Punitha, K.A.J.; Ramaraj, N. Test case generation and prioritization for composite web service based on owl-s. Neural Netw. World 2011, 21, 519–538.
  19. Askarunisa, A.; Punitha, K.A.J.; Abirami, A.M. Black box test case prioritization techniques for semantic based composite web services using OWL-S. In Proceedings of the 2011 International Conference on Recent Trends in Information Technology (ICRTIT), Chennai, India, 3–5 June 2011; pp. 1215–1220.
  20. Bai, X.; Kenett, R.S.; Yu, W. Risk assessment and adaptive group testing of semantic web services. Int. J. Softw. Eng. Knowl. Eng. 2012, 22, 595–620.
  21. Kim, M.; Cobb, J.; Harrold, M.J.; Kurc, T.; Orso, A.; Saltz, J.; Post, A.; Malhotra, K.; Navathe, S.B. Efficient regression testing of ontology-driven systems. In Proceedings of the ISSTA 2012 International Symposium on Software Testing and Analysis, Minneapolis, MN, USA, 15–20 July 2012; pp. 320–330.
  22. Fan, C.-F.; Wang, W.-S. Validation test case generation based on safety analysis ontology. Ann. Nucl. Energy 2012, 45, 46–58.
  23. Tseng, W.-H.; Fan, C.-F. Systematic scenario test case generation for nuclear safety systems. Inf. Softw. Technol. 2013, 55, 344–356.
  24. Ali, S.; Yue, T.; Hoffmann, A.; Wendland, M.-F.; Bagnato, A.; Brosse, E.; Schacher, M.; Dai, Z.R. How Does the UML Testing Profile Support Risk-Based Testing. In Proceedings of the 2014 IEEE International Symposium on Software Reliability Engineering Workshops, Naples, Italy, 3–6 November 2014; pp. 311–316.
  25. Li, Y.; Tao, J.; Wotawa, F. Ontology-based test generation for automated and autonomous driving functions. Inf. Softw. Technol. 2020, 117, 106200.
  26. Zhu, H.; Zhang, Y. Collaborative Testing of Web Services. IEEE Trans. Serv. Comput. 2010, 5, 116–130.
  27. Mei, L.; Chan, W.K.; Tse, T.H.; Jiang, B.; Zhai, K. Preemptive regression testing of workflow-based web services. IEEE Trans. Serv. Comput. 2014, 8, 740–754.
  28. Hillah, L.M.; Maesano, A.-P.; De Rosa, F.; Kordon, F.; Wuillemin, P.-H.; Fontanelli, R.; Di Bona, S.; Guerri, D.; Maesano, L. Automation and intelligent scheduling of distributed system functional testing. Int. J. Softw. Tools Technol. Transf. 2017, 19, 281–308.
  29. Mani, P.; Prasanna, M. Validation of automated test cases with specification path. J. Stat. Manag. Syst. 2017, 20, 535–542.
More