Performance Comparison of Automated Website GUI Testing Tools: A Study of Selenium IDE, Katalon Studio, UI.Vision, and BugBug
##plugins.themes.bootstrap3.article.sidebar##
##plugins.themes.bootstrap3.article.main##
Muhammad Javier Dafa Rozi
Endang Sulistiyani
Rizqi Putri Nourma Budiarti
Abstract
Software testing consumes approximately 50% of development time and cost. One of the most commonly used testing activities is GUI testing, which is still frequently carried out manually. In manual testing, testers often repeat the same actions multiple times, increasing the risk of human error. In contrast, automated testing utilizes specialized tools to minimize these errors. Therefore, selecting the right tool and understanding its performance are crucial. Previous studies on automated testing generally focused only on implementing test objects without evaluating the actual performance quality of the tools used. There are many automated testing tools available that offer capture-and-replay features, such as Selenium, Katalon Studio, UI.Vision, and BugBug. This study compares the performance of these tools for website GUI testing using two parameters: execution time and encountered issues. A total of 21 test cases were designed and executed by the researcher. The execution results show that Katalon Studio, UI.Vision, and BugBug successfully completed all test cases as expected. However, Selenium failed to execute 5 test cases due to its inability to perform hover dropdown actions. In contrast, Katalon Studio, UI.Vision, and BugBug did not encounter issues during execution. In terms of test execution time, Katalon Studio recorded the fastest average time at 6.15 seconds, followed by Selenium. UI.Vision and BugBug required longer execution times, with average times of 20.85 seconds and 22.8 seconds respectively.
Downloads
##plugins.themes.bootstrap3.article.details##
Copyright (c) 2025 Muhammad Javier Dafa Rozi, Endang Sulistiyani, Rizqi Putri Nourma Budiarti
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
References
H. V. Gamido and M. V. Gamido, “Comparative review of the features of automated software testing tools,” Int. J. Electr. Comput. Eng., vol. 9, no. 5, pp. 4473–4478, 2019, doi: 10.11591/ijece.v9i5.pp4473-4478.
D. Setiawan, M. A. Fadhillah, A. Wibawa, I. Sugiarto, A. Mulyana, and I. Kusyadi, “Pengujian Black Box pada Aplikasi Perpustakaan Berbasis Web Menggunakan Teknik Equivalence Partitioning,” J. Teknol. Sist. Inf. dan Apl., vol. 3, no. 2, p. 95, 2020, doi: 10.32493/jtsi.v3i2.3955.
M. T. Abdillah et al., “Implementasi Black box Testing dan Usability Testing pada Website Sekolah MI Miftahul Ulum Warugunung Surabaya,” J. Ilmu Komput. dan Desain Komun. Vis., vol. 8, no. 1, 2023.
M. Swillus and A. Zaidman, “Sentiment overflow in the testing stack: Analyzing software testing posts on Stack Overflow,” J. Syst. Softw., vol. 205, p. 111804, 2023, doi: 10.1016/j.jss.2023.111804.
H. Rusli, “Analisa perbandingan black-box automated testing dan manual testing pada aplikasi accmart,” 2020.
A. Arcuri, “Automated Black- And White-Box Testing of RESTful APIs with EvoMaster,” IEEE Softw., vol. 38, no. 3, pp. 72–78, 2021, doi: 10.1109/MS.2020.3013820.
N. U. Hasanah, Kajian otomatisasi pengujian gui: selenium ide, Uipath Studio, Katalon studio. 2022. [Online]. Available: https://repository.uinjkt.ac.id/dspace/handle/123456789/66772%0Ahttps://repository.uinjkt.ac.id/dspace/bitstream/123456789/66772/1/NURUL USWATUN HASANAH-FST.pdf
M. Cernat, A. N. Staicu, and A. Stefanescu, “Improving UI test automation using robotic process automation,” ICSOFT 2020 - Proc. 15th Int. Conf. Softw. Technol., no. Icsoft, pp. 260–272, 2020, doi: 10.5220/0009911202600267.
M. M. Muhtadi, M. D. Friyadi, and A. Rahmani, “Analisis GUI Testing pada Aplikasi E-Commerce menggunakan Katalon,” Pros. Ind. Res. Work. Natl. Semin., vol. 10, no. 1, pp. 1387–1393, 2019.
S. Melia and F. P. Putra, “Comparative Analysis of Automated Testing Tools on GUI WEB-Based Applications Analisis Perbandingan Tools Pengujian Otomatis pada GUI Aplikasi Berbasis WEB,” pp. 267–273, 2023.
M. Leotta, D. Clerissi, F. Ricca, and P. Tonella, Approaches and Tools for Automated End-to-End Web Testing, 1st ed., vol. 101. Elsevier Inc., 2016. doi: 10.1016/bs.adcom.2015.11.007.
H. Anjum et al., “Otomatisasi Pengujian Aplikasi Web Toko Sembako Menggunakan Selenium IDE,” Log. J. Ilmu Komput. dan Pendidik., vol. 1, no. 2, pp. 303–309, 2023, [Online]. Available: https://journal.mediapublikasi.id/index.php/logic/article/view/1654
R. P. Octavially, R. R. Riskiana, K. A. Laksitowening, D. S. Kusumo, M. Adrian, and N. Selviandro, “Test Case Analysis with Keyword-Driven Testing Approach on Angkasa Website Using Katalon Studio Tools,” Ultim. J. Tek. Inform., vol. 13, no. 2, pp. 134–141, 2022, doi: 10.31937/ti.v13i2.2391.
E. Pelivani and B. Cico, “A comparative study of automation testing tools for web applications,” 2021 10th Mediterr. Conf. Embed. Comput. MECO 2021, pp. 7–10, 2021, doi: 10.1109/MECO52532.2021.9460242.
Meenu and Y. Kumar, “Comparative Study of Automated Testing Tools: Selenium , SoapUI, HP Unified Functional Testing and Test Complete,” J. Emerg. Technol. Innov. Res., vol. 2, no. 9, pp. 42–48, 2015, [Online]. Available: www.jetir.org
S. S. Rahayu et al., “Analisis Penggunaan Tools Automation Testing pada Aplikasi : Systematic Literature Review,” vol. 8, pp. 106–116, 2024.
B. Tech, “OVERVIEW AND ANALYSIS OF AUTOMATED TESTING TOOLS: RANOREX, TEST COMPLETE, SELENIUM DHEERAJ KAKARAPARTHY IRJET Journal OVERVIEW AND ANALYSIS OF AUTOMATED TESTING TOOLS: RANOREX, TEST COMPLETE, SELENIUM DHEERAJ KAKARAPARTHY,” Int. Res. J. Eng. Technol., pp. 1575–1579, 2017, [Online]. Available: www.irjet.net
A. C. Barus and S. Leo, “Android Comparative Study of Automated Testing Tools for Android,” JTIIK J. Teknol. Inf. dan Ilmu Komput., vol. 6, no. 6, pp. 645–654, 2019, doi: 10.25126/jtiik.20196953.
J. Wang, Y. Wang, and S. Cheng, “A survey on GUI testing techniques for mobile applications,” IEEE Access, vol. 9, pp. 4533–4548, 2021, doi: 10.1109/ACCESS.2021.3049440.
A. M. Memon, “Automatically repairing event sequence-based GUI test suites for regression testing,” ACM Trans. Softw. Eng. Methodol., vol. 29, no. 4, pp. 1–35, 2020, doi: 10.1145/3380943.
M. Leotta, D. Clerissi, F. Ricca, and P. Tonella, “Capture-replay tools for mobile and web testing: A systematic review,” Softw. Test. Verif. Reliab., vol. 32, no. 1, pp. 1–34, 2022, doi: 10.1002/stvr.1763.
Y. Li, C. Xu, and L. Zhang, “DeepTest: Automated testing for deep-learning-based autonomous systems,” Proc. IEEE Int. Conf. Softw. Eng., pp. 101–112, 2021, doi: 10.1109/ICSE43902.2021.00020.
A. Anand et al., “Efficient GUI testing using model-based and machine-learning approaches,” IEEE Trans. Softw. Eng., vol. 48, no. 3, pp. 789–803, 2022, doi: 10.1109/TSE.2020.3048802.
L. K. Sharif, S. S. Idrus, and A. A. Rahman, “Evaluating Selenium WebDriver performance using multiple browsers,” Indones. J. Electr. Eng. Comput. Sci., vol. 24, no. 2, pp. 998–1007, 2021, doi: 10.11591/ijeecs.v24.i2.pp998-1007.
M. Gutiérrez et al., “Test automation for web applications: A review of tools, techniques, and challenges,” J. Syst. Softw., vol. 190, p. 111327, 2022, doi: 10.1016/j.jss.2022.111327.
P. N. Robles, F. P. Miller, and R. L. Smith, “Assessing maintainability problems in GUI test scripts,” IEEE Softw., vol. 37, no. 4, pp. 62–71, 2020, doi: 10.1109/MS.2020.2974979.
A. O. Olanrewaju and J. A. Adu, “Comparison of web test automation frameworks: Selenium, Cypress, and Playwright,” IEEE Africon, pp. 812–817, 2023, doi: 10.1109/AFRICON55921.2023.10247652.
C. Pinto, R. Dias, and A. Sampaio, “Model-driven automation for GUI testing in web applications,” IEEE Int. Symp. Softw. Reliab. Eng., pp. 55–66, 2021, doi: 10.1109/ISSRE52982.2021.00018