The Theory of Mobile Usability

Version	Summary	Created by	Modification	Content Size	Created at	Operation
1		Paweł Weichbroth	--	1525	2024-02-29 15:28:56	\|
2	layout	Jessie Wu	-1 word(s)	1524	2024-03-01 04:23:03	\|

This entry is adapted from the peer-reviewed paper 10.3390/app14051792

The success of a new mobile application depends on a variety of factors ranging from business understanding, customer value, and perceived quality of use. In this sense, the topic of usability testing of mobile applications is relevant from the point of view of user satisfaction and acceptance.

mobile usability testing methodology framework

1. Usability Conceptualization

In light of a recent study ^[1], the most widely accepted definition of usability is that provided in the ISO 9241-11 standard, which states that usability is “the extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness efficiency and satisfaction in a specified context of use” ^[2].

At this point, of an obvious nature, a question arises: how does one understand context? Context can be understood as a carrier of information about the environment, place, time, and situation in which an entity currently exists ^[3]. Here, an entity is a user who deliberately interacts with a mobile application. With the ubiquity of mobile devices (GPS devices) ^[4] and Internet connectivity (public Wi-Fi hotspots, home Wi-Fi, LTE 4G, and 5G) ^[5], the ability to incorporate this type of information is common and in many domains has become even an imperative to use ^[6]. In summary, context in mobile systems can be divided into three categories ^[7]:

external, independent, and valid for all interested users (e.g., current weather, dangerous events, time, etc.);
location, refers to information about the user’s point of interest (e.g., traffic jams, road conditions, parking space, restaurant reviews, etc.);
user-specific, related to the user’s attributes, beliefs, activities, and interests (e.g., gender, age, nationality, religion, etc.).

Incorporating such context in mobile applications significantly enhances the quality of service in terms of perceived usefulness by making our everyday environments increasingly intelligent ^[8].

2. Usability Attributes Conceptualization

By definition, an attribute is a quality or feature regarded as a characteristic or inherent part of something ^[9]. Similarly to the notion of usability, attributes do not exist as such. On the contrary, they emerge from the physical interaction between the user and the mobile application. If now one takes into account the aforementioned usability definition, the question arises as to how to measure the extent of effectiveness, efficiency, and satisfaction. The answer is twofold: through user observation or by user survey.

That being said, an attribute can be classified as “observable” or as “perceived”, respectively. While it is possible to change the type from the former to the latter, then the reverse operation is hardly achievable, or even impossible, due to human nature. For instance, very few users, if any, explicitly manifest satisfaction during or after using typical mobile applications. Nevertheless, there have been attempts to identify, measure, and evaluate numerous qualities with regard to both the user and the application, especially in domains such as games ^[10] or entertainment ^[11]^[12].

Let us now look at three attributes referred to in the ISO 9241-11 standard. It should be noted that, while effectiveness and efficiency are directly observable qualities, satisfaction is a “hidden” quality. Moreover, it is also possible to measure both effectiveness and efficiency through user survey. In short, Table 1 shows the 2-category classification of the ISO 9241-11 usability attributes.

Table 1. Usability attributes classification.

Attribute/Type	Observed	Perceived
Effectiveness	•	•
Efficiency	•	•
Satisfaction	Not applicable	•

Such a distinction has implications for the conceptualization of the usability attributes. Firstly, in the case of the observed category, the object of measurement is a user, or, more precisely, the user’s level of task performance. With this assumption, Table 2 shows the definitions of the observed usability attributes.

Table 2. The conceptualization of the observed usability attributes.

Attribute	Definition
Effectiveness	the ability of a user to complete a task in a given context ^[13]
Efficiency	the ability of a user to complete a task with speed and accuracy ^[14]
Satisfaction	Not applicable

Secondly, in the case of the second category, the object of measurement is a mobile application, in particular the user’s perceived level of workload and application performance, as well as the self-reported level of satisfaction. The definitions of the perceived usability attributes are provided in Table 3.

Table 3. The conceptualization of the perceived usability attributes.

Attribute	Definition
Effectiveness	a user’s perceived level of workload in a given context ^[15]
Efficiency	a user’s perceived level of application performance (in terms of time) ^[16]
Satisfaction	a user’s perceived level of comfort and pleasure ^[17]

In summary, the observed attributes can be interpreted in terms of the performance-based characteristics of the user, whereas the perceived attributes can be interpreted in terms of the user’s perceptions of certain application characteristics, as well as their own feelings of comfort and task fulfilment.

It should also be noted that there are other commonly studied attributes that are considered latent variables. In this regard, the most frequent ones also concern ^[1] learnability, memorability, cognitive load, simplicity, and ease of use.

3. Usability Attributes Operationalization

By definition, operationalization is “the process by which a researcher defines how a concept is measured, observed, or manipulated within a particular study” ^[18]. More specifically, the researcher translates the conceptual variable of interest into a set of specific “measures” ^[19]. Note that, here, a measure is a noun and means a way of measuring with the units used for stating the particular property (e.g., size, weight, and time), whereas “measures of quantitative assessment commonly used for assessing, comparing, and tracking performance or production” are termed as metrics ^[20]. In other words, a metric is a quantifiable measure of the observed variable.

However, the other way to quantify variables is to use indicators. By definition, an indicator is “a quantitative or qualitative variable that provides reliable means to measure a particular phenomenon or attribute” ^[21]. Indicators are used to operationalize latent variables ^[22], in both reflective and formative measurement models ^[23]. In summary, for the sake of methodological clarity of the above terms “metric” and “indicator”, only the former will be used for both observable and perceived attributes.

Drawing upon the usability attributes classification, now researchers can turn to operationalize them, which requires specification of the quantifiable metrics, along with corresponding measurement scales.

3.1. Observed Effectiveness

To quantify the observed effectiveness of a user in the context of the performed tasks, in total, five metrics are provided in Table 4 with assigned units and quantities.

Table 4. The observed effectiveness metrics.

Code	Metric	Unit and Quantity
EFFE1	rate of successful task completion	integer/amount
EFFE2	total number of steps required to complete a task	integer/amount
EFFE3	total number of taps related to app usage	integer/amount
EFFE4	total number of taps unrelated to app usage	integer/amount
EFFE5	total number of times that the back button was used	integer/amount

3.2. Observed Efficiency

By definition, efficiency is a quality that is measured by the amount of resources that are used by a mobile application to produce a given number of outputs. Now, thinking in terms of usability testing, the measured resource concerns the amount of time that a user needed to perform a particular task. Thus, the observed efficiency is measured by the completion time (EFFI1 metric) in units of time (commonly in seconds) with respect to each individual task, or much less often to a set of related tasks.

3.3. Perceived Effectiveness

It should be noted that observed and perceived effectiveness are measured by the same metrics except for the first one (EFFE1) since its submission to the respondent would imply a self-assessment of the rate of task completion. The following 7-point Likert scale can be used: absolutely inappropriate (1), inappropriate (2), slightly inappropriate (3), neutral (4), slightly appropriate (5), appropriate (6), and absolutely appropriate (7).

3.4. Perceived Efficiency

If researchers consider efficiency as an unobservable construct, the 7-point rating scale is also used to measure and rate the mobile application in this view. Table 5 shows the details of the perceived efficiency metrics.

Table 5. The perceived efficiency metrics ^[24].

Code	Metric	Scale
EFFI2	duration of the application starting *	7-point Likert scale
EFFI3	duration of the application closing *
EFFI4	duration of content loading *
EFFI5	duration of the application response to the performed actions *
EFFI6	application performance continuity

* the reverse scale.

Similarly, if efficiency is treated as an unobservable construct, the 7-point Likert rating scale can be used to measure and evaluate the mobile application in this perspective, starting from extremely low (1), very low (2), low (3), moderate (4), high (5), very high (6), to extremely high (7). Note that, for all metrics, expect the last one, a reverse scale must be used to estimate the perceived efficiency in order to preserve the correct interpretation of the collected data.

3.5. Perceived Satisfaction

In general, satisfaction is “a pleasant feeling you get when you get something you wanted or when you have done something you wanted to do” ^[25]. The perceived satisfaction construct (SATI) is composed of the three metrics validated in other usability studies. Table 6 provides a detailed description.

Table 6. The perceived satisfaction metrics.

Code	Metric	Scale
SATI1	I think I made the correct decision to use the X mobile application ^[26]	7-point Likert scale
SATI2	My experience using X mobile app has been satisfactory ^[27]
SATI3	I am satisfied with the quality of X ^[28]

X is the name of the mobile application being evaluated.

The following 7-point Likert scale can be used, starting with strongly disagree (1), disagree (2), somewhat disagree (3), neither agree nor disagree (4), somewhat agree (5), agree (6), and strongly agree (7).

References

Weichbroth, P. Usability of mobile applications: A systematic literature study. IEEE Access 2020, 8, 55563–55577.
ISO 9241-11:2018; Ergonomics of Human-System Interaction—Part 11: Usability: Definitions and Concepts. International Organization for Standardization: Geneva, Switzerland, 2018.
Owoc, M.; Weichbroth, P.; Żuralski, K. Towards better understanding of context-aware knowledge transformation. In Proceedings of the 2017 Federated Conference on Computer Science and Information Systems (FedCSIS), Prague, Czech Republic, 3–6 September 2017; IEEE: Piscataway, NJ, USA, 2017; pp. 1123–1126.
Yuan, H.; Jin, T.; Ye, X. Establishment and Application of Crowd-Sensing-Based System for Bridge Structural Crack Detection. Appl. Sci. 2023, 13, 8281.
Rafique, U.; Khan, S.; Ahmed, M.M.; Kiani, S.H.; Abbas, S.M.; Saeed, S.I.; Alibakhshikenari, M.; Dalarsson, M. Uni-planar MIMO antenna for sub-6 GHz 5G mobile phone applications. Appl. Sci. 2022, 12, 3746.
Nakhimovsky, Y.; Miller, A.T.; Dimopoulos, T.; Siliski, M. Behind the scenes of google maps navigation: Enabling actionable user feedback at scale. In Proceedings of the CHI’10 Extended Abstracts on Human Factors in Computing Systems, Atlanta, CA, USA, 10–15 April 2010; pp. 3763–3768.
Musumba, G.W.; Nyongesa, H.O. Context awareness in mobile computing: A review. Int. J. Mach. Learn. Appl. 2013, 2, 5.
Luo, C.; Goncalves, J.; Velloso, E.; Kostakos, V. A survey of context simulation for testing mobile context-aware applications. ACM Comput. Surv. (CSUR) 2020, 53, 1–39.
Encyclopedia.com. Attribute. 2024. Available online: https://www.encyclopedia.com/science-and-technology/computers-and-electrical-engineering/computers-and-computing/attribute (accessed on 20 January 2024).
Zaibon, S.B. User testing on game usability, mobility, playability, and learning content of mobile game-based learning. J. Teknol. 2015, 77, 131–139.
Gilbert, A.L.; Sangwan, S.; Ian, H.H.M. Beyond usability: The OoBE dynamics of mobile data services markets. Pers. Ubiquitous Comput. 2005, 9, 198–208.
Silvennoinen, J.; Vogel, M.; Kujala, S. Experiencing visual usability and aesthetics in two mobile application contexts. J. Usability Stud. 2014, 10, 46–62.
Widyanti, A.; Ainizzamani, S.A.Q. Usability evaluation of online transportation’user interface. In Proceedings of the 2017 International Conference on Information Technology Systems and Innovation (ICITSI), Bandung, Indonesia, 23–24 October 2017; IEEE: Piscataway, NJ, USA, 2017; pp. 82–86.
Harrison, R.; Flood, D.; Duce, D. Usability of mobile applications: Literature review and rationale for a new usability model. J. Interact. Sci. 2013, 1, 1–16.
John, B.E.; Marks, S.J. Tracking the effectiveness of usability evaluation methods. Behav. Inf. Technol. 1997, 16, 188–202.
Jeng, J. Usability assessment of academic digital libraries: Effectiveness, efficiency, satisfaction, and learnability. Libri 2005, 55, 96–121.
Kabir, M.A.; Salem, O.A.; Rehman, M.U. Discovering knowledge from mobile application users for usability improvement: A fuzzy association rule mining approach. In Proceedings of the 2017 8th IEEE International Conference on Software Engineering and Service Science (ICSESS), Beijing, China, 24–26 November 2017; IEEE: Piscataway, NJ, USA, 2017; pp. 126–129.
Baumeister, R.F. Encyclopedia of Social Psychology; Sage: London, UK, 2007; Volume 1.
Choi, W.; Stvilia, B. Web credibility assessment: Conceptualization, operationalization, variability, and models. J. Assoc. Inf. Sci. Technol. 2015, 66, 2399–2414.
Carneiro, A. Maturity and metrics in health organizations information systems. In Handbook of Research on ICTs and Management Systems for Improving Efficiency in Healthcare and Social Care; IGI Global: Hershey, PA, USA, 2013; pp. 937–952.
United States Agency for International Development. Glossary of Evaluation Terms. 2009. Available online: https://pdf.usaid.gov/pdf_docs/PNADO820.pdf (accessed on 20 January 2024).
Bollen, K.A.; Diamantopoulos, A. In defense of causal-formative indicators: A minority report. Psychol. Methods 2017, 22, 581.
Edwards, J.R.; Bagozzi, R.P. On the nature and direction of relationships between constructs and measures. Psychol. Methods 2000, 5, 155.
Weichbroth, P. An empirical study on the impact of gender on mobile applications usability. IEEE Access 2022, 10, 119419–119436.
Cambridge Dictionary. Satisfaction. 2024. Available online: https://dictionary.cambridge.org/dictionary/english/satisfaction (accessed on 20 January 2024).
Liébana-Cabanillas, F.; Molinillo, S.; Ruiz-Montañez, M. To use or not to use, that is the question: Analysis of the determining factors for using NFC mobile payment systems in public transportation. Technol. Forecast. Soc. Chang. 2019, 139, 266–276.
Hsiao, C.H.; Chang, J.J.; Tang, K.Y. Exploring the influential factors in continuance usage of mobile social Apps: Satisfaction, habit, and customer value perspectives. Telemat. Inform. 2016, 33, 342–355.
Wan, L.; Xie, S.; Shu, A. Toward an understanding of university students’ continued intention to use MOOCs: When UTAUT model meets TTF model. Sage Open 2020, 10, 2158244020941858.

© Text is available under the terms and conditions of the Creative Commons Attribution (CC BY) license; additional terms may apply. By using this site, you agree to the Terms and Conditions and Privacy Policy.

Upload a video for this entry

Information

Subjects: Computer Science, Information Systems

Contributor MDPI registered users' name will be linked to their SciProfiles pages. To register with us, please refer to https://encyclopedia.pub/register :

Paweł Weichbroth

View Times: 71

Update Date: 01 Mar 2024

Table of Contents

Video Upload Options

Confirm