Expert Review on Usefulness of an Integrated Checklist-based Mobile Usability Evaluation Framework

Authors

  • Hazura Zulzalil

    Department of Software Engineering and Information System, Universiti Putra Malaysia, UPM Serdang, Selangor, 43400, Malaysia
  • Hazwani Rahmat

    Department of Information Technology, Centre for Diploma Studies, Universiti Tun Hussein Onn Malaysia, Pagoh Higher Education Hub, Pagoh, Johor, 84600, Malaysia
  • Abdul Azim Abd Ghani

    Department of Software Engineering and Information System, Universiti Putra Malaysia, UPM Serdang, Selangor, 43400, Malaysia
  • Azrina Kamaruddin

    Department of Software Engineering and Information System, Universiti Putra Malaysia, UPM Serdang, Selangor, 43400, Malaysia

DOI:

https://doi.org/10.30564/jcsr.v5i3.5816
Received: 28 June 2023 | Revised: 24 July 2023 | Accepted: 26 July 2023 | Published Online: 31 July 2023

Abstract

Previous mobile usability studies are only pertinent in the context of ergonomics, physical user interface, and mobility aspects. In addition, much of the previous mobile usability conception was built on desktop computing measurements, such as desktop and web application checklists, or scarcely addressed the mobile user interface. Moreover, the studies focus mainly on interface features for desktop applications and do not reflect comprehensive mobile interface features such as navigation drawers and spinners. Therefore, conducting usability evaluation using conventional usability measurement would result in irrelevant results. In addition, the resulting works are tailored for usability testing, which requires highly skilled evaluators and usability specialists (e.g., usability testers and user experience designers), who are rarely integrated into a development team. The lack of expertise could lead to unreliable usability evaluations. This paper presents a review from industrial experts on a comprehensive and feasible usability evaluation framework developed in our previous work. The framework is dedicated to smartphone apps, which integrate evaluator skills and design concerns. However, there is no evidence of its usefulness in practice. Therefore, the usefulness of the framework measurement for evaluating apps' usability in the eyes of non-usability specialists is empirically assessed in this paper through an expert review. The expert review involved eleven industrial developers and was complemented by a semi-structured interview. The method is replicated in comparison with a framework from another study. The findings show that the formulated framework significantly outperformed the framework (p = 0.0286) from other studies with large effect sizes (r = 1.81) in terms of usefulness.

Keywords:

Usability framework, Mobile usability, Usability evaluation, Expert review, Heuristic walkthrough

References

[1] Moumane, K., Idri, A., Abran, A., 2016. Usability evaluation of mobile applications using ISO 9241 and ISO 25062 standards. SpringerPlus. 5, 1-15.

[2] Hussain, A., Abubakar, H.I., Hashim, N.B. (editors), 2014. Evaluating mobile banking application: Usability dimensions and measurements. Proceedings of the 6th international Conference on Information Technology and Multimedia; 2014 Nov 18-20; Putrajaya, Malaysia. New York: IEEE. p. 136-140.

[3] Hussain, A., Kutar, M. (editors), 2009. Usability metric for mobile application. 2008 Proceedings of the 10th International Conference on Information Integration and Web-Based Applications and Services (IiWAS' 08); 2008 Nov 24-26; Linz, Austria. New York: Association for Computing Machinery. p. 567-570.

[4] Liu, N., Yu, R., 2017. Identifying design feature factors critical to acceptance and usage behavior of smartphones. Computers in Human Behavior. 70, 131-142.

[5] Harrison, R., Flood, D., Duce, D., 2013. Usability of mobile applications: literature review and rationale for a new usability model. Journal of Interaction Science. 1, 1-16.

[6] Yáñez Gómez, R., Cascado Caballero, D., Sevillano, J.L., 2014. Heuristic evaluation on mobile interfaces: A new checklist. The Scientific World Journal. 1-19.

[7] Nayebi, F., 2015. iOS application user rating prediction using usability evaluation and machine learning [Ph.D. thesis]. Quebec: University of Quebec.

[8] Saleh, A., Ismail, R., Fabil, N. (editors), 2017. Evaluating usability for mobile application: A MAUEM approach. ICSEB 2017 Proceedings of the 2017 International Conference on Software and E-Business; 2017 Dec 28-30; Hong Kong. New York: Association for Computing Machinery. p. 71-77.

[9] Dubey, S.K., Gulati, A., Rana, A., 2012. Integrated model for software usability. International Journal on Computer Science and Engineering. 4(3), 429.

[10] Elsantil, Y., 2020. User perceptions of the security of mobile applications. International Journal of E-Services and Mobile Applications (IJESMA). 12(4), 24-41. DOI: https://doi.org/10.4018/IJESMA.2020100102

[11] Malatini, S., Bogliolo, A. (editors), 2015. Gamification in mobile applications usability evaluation: A New Approach. MobileHCI' 15: Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services; 2015 Aug 24-27; Copenhagen, Denmark. New York: Association for Computing Machinery. p. 897-899.

[12] Hoehle, H., Aljafari, R., Venkatesh, V., 2016. Leveraging Microsoft's mobile usability guidelines: Conceptualizing and developing scales for mobile application usability. International Journal of Human-Computer Studies. 89, 35-53.

[13] Mugisha, A., Nankabirwa, V., Tylleskär, T., et al., 2019. A usability design checklist for Mobile electronic data capturing forms: the validation process. BMC Medical Informatics and Decision Making. 19(1), 1-11. DOI: https://doi.org/10.1186/S12911-018-0718-3

[14] Xu, H., Jonsson, M., 2012. Tablet application GUI usability checklist—Creation of a user interface usability checklist for tablet applications [Master's thesis]. Huddinge: Södertörns University College.

[15] Lachgar, M., Abdelmounaim, A., 2017. Decision framework for mobile development methods. International Journal of Advanced Computer Science and Applications. 8(2), 110-118. DOI: https://doi.org/10.14569/IJACSA.2017.080215

[16] ISO 9241-11:1998 Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs)—Part 11: Guidance on Usability [Internet]. International Organization for Standardization; 1998 [cited 2018 Dec 21]. Available from: https://www.iso.org/standard/16883.html

[17] ISO/IEC 9126-1:2001 Software Engineering—Product Quality—Part 1: Quality Model [Internet]. International Standard for Standardization; 2001 [cited 2018 Dec 21]. Available from: https://www.iso.org/standard/22749.html

[18] Nielsen, J., Budiu, R., 2012. Mobile usability. New Riders Press: Berkeley CA.

[19] Constantine, L.L., Lockwood, L.A., 1999. Software for use: A practical guide to the models and methods of usage-centered design. Addison-Wesley Publishing Co.: Boston.

[20] Böckle, M., Rühmkorf, J. (editors), 2019. Towards a framework for the classification of usability issues. Human-Computer Interaction–INTERACT 2019: 17th IFIP TC 13 International Conference; 2019 Sep 2-6; Paphos, Cyprus. p. 610-614.

[21] Kronbauer, A.H., Santos, C.A., Vieira, V. (editors), 2012. Smartphone applications usability evaluation: A hybrid model and its implementation. Human-Centered Software Engineering: 4th International Conference, HCSE 2012; 2012 Oct 29-31; Toulouse, France. p. 146-163.

[22] Olsina, L., Santos, L., Lew, P., 2014. Evaluating mobileapp usability: A holistic quality approach. Lecture notes in computer science. Springer, Cham.: New York. pp. 111-129.

[23] Fabil, N.B., Saleh, A., Isamil, R.B., 2015. Extension of pacmad model for usability evaluation metrics using goal question metrics (Gqm) approach. Journal of Theoretical and Applied Information Technology. 79(1), 90-100.

[24] Inostroza, R., Rusu, C., Roncagliolo, S., et al. (editors), 2012. Usability heuristics for touchscreen-based mobile devices. 2012 Ninth International Conference on Information Technology—New Generations; 2012 Apr 16-18; Las Vegas, NV, USA. New York: IEEE.

[25] Mi, N., Cavuoto, L.A., Benson, K., et al., 2014. A heuristic checklist for an accessible smartphone interface design. Universal Access in the Information Society. 13, 351-365.

[26] Soomro, S., Ahmad, W.F.W., Sulaiman, S. (editors), 2012. A preliminary study on heuristics for mobile games. 2012 International Conference on Computer and Information Science; 2012 Jun 12-14; Kuala Lumpur, Malaysia. New York: IEEE. p. 1030-1035.

[27] Zahra, F., Hussain, A., Mohd, H. (editors), 2017. Usability evaluation of mobile applications; Where do we stand? The 2nd International Conference on Applied Science and Technology 2017 (ICAST'17); 2017 Apr 3-5; Kedah, Malaysia.

[28] Zamfiroiu, A., 2014. Factors influencing the quality of mobile applications. Informatica Economica. 18(1), 131.

[29] Homann, M., Wittges, H., Krcmar, H., 2013. Towards user interface patterns for ERP applications on smartphones. Business Information Systems. 157, 14-25.

[30] Roder, H. (editor), 2012. Specifying usability features with patterns and templates. 2012 First International Workshop on Usability and Accessibility Focused Requirements Engineering (UsARE); 2012 Jun 28; Zurich, Switzerland. New York: IEEE. p. 6-11.

[31] Punchoojit, L., Hongwarittorrn, N., 2017. Usability studies on mobile user interface design patterns: A systematic literature review. Advances in Human-Computer Interaction. 1-22. DOI: https://doi.org/10.1155/2017/6787504

[32] Rahmat, H., Zulzalil, H., Ghani, A.A.A., et al., 2018. A comprehensive usability model for evaluating smartphone apps. Advanced Science Letters. 24(3), 1633-1637.

[33] Zulzalil, H., Rahmat, H., Ghani, A.A.A., et al., 2019. Conceptualising mobile apps usability dimension: A feasibility assessment of Malaysian industrial practitioners. International Journal of Engineering and Advanced Technology. 9(1), 1708-1713.

[34] Tarkkanen, K., Harkke, V., Reijonen, P., 2015. Are we testing utility? Analysis of usability problem types. Design, user experience, and usability: Design discourse. Springer: Cham. pp. 269-280. DOI: https://doi.org/10.1007/978-3-319-20886-2_26

[35] MacDonald, C.M., Atwood, M.E. (editors), 2014. What does it mean for a system to be useful? Proceedings of the 2014 Conference on Designing Interactive Systems; 2014 Jun 21-25; Vancouver BC, Canada. New York: Association for Computing Machinery. p. 885-894.

[36] Joseph, V., 2017. User experience guidelines for improving retention rate in mobile apps [Master's thesis]. Madrid: Universidad Politécnica de Madrid.

[37] Thitichaimongkhol, K., Senivongse, T., 2016. Enhancing usability heuristics for android applications on mobile devices. Proceedings of the World Congress on Engineering and Computer Science. 1, 19-21.

[38] Sullivan, G.M., Feinn, R., 2012. Using effect size—or why the P value is not enough. Journal of Graduate Medical Education. 4(3), 279-282.

Downloads

How to Cite

Zulzalil, H., Rahmat, H., Ghani, A. A. A., & Kamaruddin, A. (2023). Expert Review on Usefulness of an Integrated Checklist-based Mobile Usability Evaluation Framework. Journal of Computer Science Research, 5(3), 57–73. https://doi.org/10.30564/jcsr.v5i3.5816

Issue

Vol. 5 , Iss. 3 (July 2023)

Article Type

Article

License

Copyright © 2023 Author(s)

Creative Commons License

This is an open access article under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) License.

Crossref
Scopus
Google Scholar
Europe PMC