Usability Evaluations are not reliable.
In his seminal work Against method: Outline of an anarchistic theory of knowledge Paul Feyerabend (Feyerabend, 1975) examines major scientific breakthroughs in the world and concludes that none of them resulted from following any explicit, formal or central method (Woolrych, Hornbæk, Frøkjær, & Cockton, 2011). Conforming to the rigors of any method does not guarantee results. Industry and academia have seen a major shift of emphasis to UX bringing usability in central focus of product (software, hardware, system, service, website, application) design and development. A plethora of usability evaluation methods exist and are in use –researchers have studied almost dozens of methods and provided overviews of development needs of these methods (Vermeeren, Law, & Roto, 2010), but are these methods reliable enough to drive future success of the product? Studies have shown that many products, even after employing many Usability Evaluation Methods (UEM’s), fail to meet customer expectations and have resulted in many organizations folding shop (Fernandez, Insfran, & Abrahão, 2008). This paper will begin by examining the reliability of usability testing, present factors that affect reliability and critique literature on those factors and conclude with a recommendation on steps that can be taken to make UEM’s more reliable.
Understanding the Problem: Factors that impact UEM’s
Suchman presented research in 1987 that showed how so called “intelligent” photocopiers built to script were no match for real user behavior and they were a miserable failure in the real world, if photocopiers were a failure in following a script then we cannot expect a complex discipline like Usability to succeed to script (...
... middle of paper ...
...itle. Plans and situated actions: The problem of human-machine communication. New York, New York, USA: Cambridge University Press.
Vermeeren, A. P. O. S., Law, E. L., & Roto, V. (2010). User Experience Evaluation Methods : Current State and Development Needs, 521–530.
Virzi, R. (1992). Refining the Test Phase of Usability Evaluation: How Many Subjects is Enough? Human Factors, 34, 457–471.
Wilson, C. E. (2006). Triangulation : The Explicit Use of Multiple Methods , Measures , and Approaches for Determining Core Issues in Product Development. Interactions, 46–48.
Woolrych, A., Hornbæk, K., Frøkjær, E., & Cockton, G. (2011). Ingredients and Meals Rather Than Recipes: A Proposal for Research That Does Not Treat Usability Evaluation Methods as Indivisible Wholes. International Journal of Human-Computer Interaction, 27(10), 940–970. doi:10.1080/10447318.2011.555314
‘The strengths, weaknesses, opportunities and threats analysis often forms the bedrock of any product planning process. It provides a simple yet effective framework for analysing both internal resources and external trends and competitors’ (Pender, L, 1999: 179)
Product managers guide new products through a complex process that includes conception, creation and commercialization (Gorchels, 2012). As this process continues, it is vital that the organization have a coherent process which it can utilize to identify products that should not continue down the development process. While there are several different processes that product managers can use to facilitate this process, this paper will focus on the stage-gate process and the critical path method.
If given a chance to do things differently, we would first include the additional factors stated above as part of the evaluation criteria of the software providers. Also, as a last step to the process, instead of only relying on the demo by the providers, we would include a ‘pilot project implementation’ stage, where the provider integrates a piece of software to the existing architecture to be used by end users in production on a daily basis. This can provide a real test of usability, compatibility and can be used to evaluate critical measures such as fault tolerance.
The Design Way lays out the fundamental principals of design forming a diagram to approach the world. Authors Harold G. Nelson, a Nierenberg Distinguished Professor of Design at Carnegie Mellon University and Erik Stolterman is Professor and Chair of Informatics at the School of Informatics and Computing, Indiana University, Bloomington (Design and Design Theory) provide an insightful look at the struggle to understand and interact with the complex world we live in. Nelson is also a Senior Instructor in the Graduate School of Business and Public Policy at the Naval Postgraduate School and President of Advanced Design Institute. Currently, Stolterman’s main work is within interaction design, philosophy and theory of design, information technology and society, information systems design, and philosophy of technology grounded in careful analytical studies of the everyday practice of users and professionals dealing with interactive artifacts with a strong emphasis of building theory. Stolterman combines this approach with a strong critical and theoretical analysis of current practice (In...
Integrated product development is based on the integrated design of products and their manufacturing and support processes. It is a matter of assessing manufacturability, reliability, and supportability of the product after it has been designed and making appropriate changes to the design to enhance these competitive factors. This approach extends the design cycle time, increases product development cost, and may not result in the most optimum way to produce the product. Instead, these factors must be considered from the very start of product development and designed into the product.
Wicked Problems in Design Thinking Author(s): Richard Buchanan Source: Design Issues, Vol. 8, No. 2 (Spring, 1992), pp. 5-21 Published by: The MIT Press Stable URL: http://www.jstor.org/
With increasing importance given to the product’s user experience in the recent days, it is essential to discuss about the reliability of usability evaluation methods. Usability evaluation methods are mainly used for identifying problems and assigning severity to them which will help the development teams in fixing the usability issues by priority. As we can see the final product usability depends a lot on the results of a usability evaluation and there has been a debate on the reliability of the evaluation methods which we have been using for long. In this paper I will be primarily discussing about the reliability of most commonly used evaluation methods (expert review, usability testing) and also discuss on the impact of evaluators experience on the usability evaluation. In the end this paper will include takeaways for the practitioner to follow in order to create increased reliability in the usability evaluation method being followed.
...ux was forty-six years.” (Zinn, 534) Unresolved issues from from as far back as the 1600’s were finally being addressed.
UCI Disabilities Services Center. (2003, February 2). Universal design and web accessibility. Retrieved September 17, 2003, fromhttp://www.disability.uci.edu/other_links/web_accessibility.htm
The organizational design, especially the particularity mentioned above, is generating a considerable amount of tension. There is a constant exchange of accusations between the two departments; marketing is critical to the product development department’s responsiveness, its time-to-market and its priorities. The counter-part was in the form of heavy critics regarding the understanding of the product development process. The constant tension only increased the rivalry and the distance between the two departments, while both should have been working as close as possible.
TurboTax is a tax preparation product for taxpayers who file quarterly estimated payments. It guides users through an interview-like process and provides payment calculators and e-filing of estimated tax payments to collect their tax return data it. Paul Coletta and Floyd Morgan, two senior employees at Intuit had worked on the TurboTax tax preparation product for several years. After working with some customers Intuit had discovered some usability issues or what they liked to call “pain points”. When attempting to fix these pain points they would either make them worse or mess up something else entirely. Coletta and Morgan felt that TurboTax needed to improve user experience and fix these pain points, they both agreed that going “social” asses that.
For this research report I decided to study a journal article that explores how computer interfaces developed to simulate natural interactions impact the overall user experience, user learning, and attitudes. This research was done by Jeeyun Oh, Harold R. Robinson, and Ji Young Lee, and was published in March 2013.
Other than that, Human interaction is also an issue on human factor social issues for using information systems in an organization. Human interaction issues include recruitment and retention of organization staffs, assessment and evaluation of users, motivation, social presence, leadership and also organizational champions. Human interaction which is also known as Human Computer Interaction or Man-machine interaction representing human and machine interface. This is because the computers or machines are totally worthless or useless unless they can be properly operated by the humans. However, both sides play an important role in a successful interaction. On the machine side, computer graphics, operating system, programming languages are an important consideration. On the other hand, the human side psychology issue such as; user satisfaction, communication theory, linguistic are brought into consideration. Human interaction is concerned with the ways humans interact with information, technologies, and tasks within various contexts. In an organization system, the system will focus on functionalities and data requirements which meet the organizational need. However, the values of the system are only visible when it is efficiently utilized by the users. Therefore, balancing between functionality and usability of a system when choosing a system is very important to achieve or accomplish goals and tasks by users. Functionality of a system is defined by the actions and services that it provides to the user. Systems are chosen based on user’s preference. Usability of a system represents the percentage of functions of the specific system are able to be used by the users. Therefore, users will choose systems which suit their usages. However,...
User participation has two distinct components: (1) user involvement which refers to subjective psychological state reflecting the importance and personal relevance that a user attached to a given system; (2) user participation which refers to the assignments, activities and behaviors that users or their representatives perform during the system development process (Tesch, D., Sobol, M., Klein, G., & Jiang, J., 2009). Hence, it is believed that the user participation has a big ...
In this evaluation of the Handbook produced by SPA we use as criteria its validity from both scientific and pedagogical point of view, its appropriateness for end users and our overall opinion about the product with relevance to the specific needs it was designed for.