Pooya Jaferian
Konstantin Beznosov
ABSTRACT
The usability of IT security management (ITSM) tools is hard to evaluate by regular methods, making heuristic evaluation attractive. However, standard usability heuristics are hard to apply as IT security management occurs within a complex and collaborative context that involves diverse stakeholders. We propose a set of ITSM usability heuristics that are based on activity theory, are supported by prior research, and consider the complex and cooperative nature of security management. In a between-subjects study, we compared the employment of the ITSM and Nielsen's heuristics for evaluation of a commercial identity management system. Participants who used the ITSM set found more problems categorized as severe than those who used Nielsen's. As evaluators identified different types of problems with the two sets of heuristics, we recommend employing both the ITSM and Nielsen's heuristics during evaluation of ITSM tools.
- K. Baker, S. Greenberg, and C. Gutwin. Heuristic evaluation of groupware based on the mechanics of collaboration. Lecture Notes in Computer Science, pages 123--140, 2001. Google Scholar
Digital Library
- K. Baker, S. Greenberg, and C. Gutwin. Empirical development of a heuristic evaluation methodology for shared workspace groupware. In Proceedings of the 2002 ACM conference on Computer supported cooperative work, CSCW '02, pages 96--105, New York, NY, USA, 2002. ACM. Google ScholarDigital Library
- L. Bauer, L. F. Cranor, R. W. Reeder, M. K. Reiter, and K. Vaniea. Real life challenges in access-control management. In CHI '09: Proceedings of the 27th international conference on Human factors in computing systems, pages 899--908, New York, NY, USA, 2009. ACM. Google ScholarDigital Library
- B. Beal. IT security: the product vendor landscape. Network Security, 2005(5):9--10, 5 2005. Google ScholarDigital Library
- D. Botta, K. Muldner, K. Hawkey, and K. Beznosov. Toward understanding distributed cognition in IT security management: the role of cues and norms. Int. Journal of Cognition, Technology & Work, Online First, September 2010. DOI: 10.1007/s10111-010-0159-y, 2010. Google ScholarCross Ref
- D. Botta, R. Werlinger, A. Gagné, K. Beznosov, L. Iverson, S. Fels, and B. Fisher. Towards understanding IT security professionals and their tools. In Proc. of Symp. On Usable Privacy and Security (SOUPS), pages 100--111, Pittsburgh, PA, July 18--20 2007. Google ScholarDigital Library
- J. M. Carroll, D. C. Neale, P. L. Isenhour, M. B. Rosson, and D. S. McCrickard. Notification and awareness: synchronizing task-oriented collaborative activity. International Journal of Human-Computer Studies, 58(5):605--632, 2003. Notification User Interfaces. Google ScholarDigital Library
- K. Charmaz. Constructing Grounded Theory. SAGE publications, 2006.Google Scholar
- P. Dourish and D. Redmiles. An approach to usable security based on event monitoring and visualization. In NSPW '02: Proceedings of the 2002 workshop on New security paradigms, pages 75--81, New York, NY, USA, 2002. ACM. Google ScholarDigital Library
- Y. Engeström. Activity theory and individual and social transformation. Perspectives on activity theory, pages 19--38, 1999.Google Scholar
- T. Erickson and W. A. Kellogg. Social translucence: an approach to designing systems that support social processes. ACM Trans. Comput.-Hum. Interact., 7(1):59--83, 2000. Google ScholarDigital Library
- G. Fitzpatrick, T. Mansfield, and S. Kaplan. Locales framework: Exploring foundations for collaboration support. In CHI'96 Sixth Australian Conference on Computer-Human Interaction, Hamilton, New Zealand, November 24--27, 34--41 1996. Google ScholarDigital Library
- A. Gagné, K. Muldner, and K. Beznosov. Identifying differences between security and other IT professionals: a qualitative analysis. In HAISA'08: Human Aspects of Information Security and Assurance, pages 69--80, Plymouth, England, July 8--9 2008.Google Scholar
- J. R. Goodall, W. G. Lutters, and A. Komlodi. I know my network: Collaboration and expertise in intrusion detection. In CSCW '04, pages 342--345, November 2004. Google ScholarDigital Library
- S. Greenberg, G. Fitzpatrick, C. Gutwin, and S. Kaplan. Adapting the locales framework for heuristic evaluation of groupware. Australian Journal of Information Systems, 7(2):102--108, 2000.Google Scholar
- C. Gutwin and S. Greenberg. The mechanics of collaboration: developing low cost usabilityevaluation methods for shared workspaces. IEEE 9th International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises, 2000. (WET ICE 2000). Proeedings, pages 98--103, 2000. Google ScholarDigital Library
- E. M. Haber and J. Bailey. Design guidelines for system administration tools developed through ethnographic field studies. In CHIMIT '07: Proceedings of the 2007 symposium on Computer human interaction for the management of information technology, pages 1:1--1:9, New York, NY, USA, 2007. ACM. Google ScholarDigital Library
- H. R. Hartson, T. S. Andre, and R. C. Williges. Criteria for evaluating usability evaluation methods. International Journal of Human-Computer Interaction, 13(4):373--410, 2001.Google ScholarCross Ref
- K. Hawkey, D. Botta, R. Werlinger, K. Muldner, A. Gagne, and K. Beznosov. Human, Organizational, and Technological Factors of IT Security. In CHI '08 extended abstract on Human factors in computing systems, pages 3639--3644, Florence, Italy, 2008. Google ScholarDigital Library
- J. Hollan, E. Hutchins, and D. Kirsh. Distributed cognition: toward a new foundation for human-computer interaction research. ACM Trans. Comput.-Hum. Interact., 7(2):174--196, 2000. Google ScholarDigital Library
- P. Jaferian, D. Botta, F. Raja, K. Hawkey, and K. Beznosov. Guidelines for Designing IT Security Management Tools. In CHIMIT '08: Proceedings of the 2008 symposium on Computer Human Interaction for the Management of Information Technology, pages 7:1--7:10. ACM, 2008. Google ScholarDigital Library
- R. Jeffries, J. R. Miller, C. Wharton, and K. Uyeda. User interface evaluation in the real world: a comparison of four techniques. In CHI '91: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 119--124, New York, NY, USA, 1991. ACM. Google ScholarDigital Library
- V. Kaptelinin and B. Nardi. Acting with technology: Activity theory and interaction design. MIT Press, 2006. Google ScholarDigital Library
- V. Kaptelinin, B. Nardi, S. Bodker, J. Carroll, J. Hollan, E. Hutchins, and T. Winograd. Post-cognitivist HCI: second-wave theories. In CHI '03: CHI '03 extended abstracts on Human factors in computing systems, pages 692--693, New York, NY, USA, 2003. ACM. Google ScholarDigital Library
- S. Kesh and P. Ratnasingam. A knowledge architecture for it security. Commun. ACM, 50(7):103--108, 2007. Google ScholarDigital Library
- A. G. Kotulic and J. G. Clark. Why there aren't more information security research studies. Information & Management, 41(5):597--607, 2004. Google ScholarDigital Library
- K. Kuutti. Activity theory as a potential framework for human-computer interaction research, pages 17--44. Massachusetts Institute of Technology, Cambridge, MA, USA, 1995. Google ScholarDigital Library
- P. P. Maglio, E. Kandogan, and E. Haber. Distributed cognition and joint activity in collaborative problem solving. In Proceedings of the Twenty-fifth Annual Conference of the Cognitive Science Society, 2003.Google Scholar
- J. Mankoff, A. K. Dey, G. Hsieh, J. Kientz, S. Lederer, and M. Ames. Heuristic evaluation of ambient displays. In Proc. CHI '03, pages 169--176, New York, NY, USA, 2003. ACM. Google ScholarDigital Library
- M. J. Muller and A. McClard. Validating an extension to participatory heuristic evaluation: quality of work and quality of work life. In CHI '95: Conference companion on Human factors in computing systems, pages 115--116, New York, NY, USA, 1995. ACM. Google ScholarDigital Library
- B. A. Nardi, editor. Context and consciousness: activity theory and human-computer interaction. Massachusetts Institute of Technology, Cambridge, MA, USA, 1995. Google ScholarDigital Library
- B. A. Nardi, S. Whittaker, and H. Schwarz. NetWORKers and their activity in intensional networks. Computer Supported Cooperative Work (CSCW), 11(1):205--242, 2002. Google ScholarDigital Library
- D. C. Neale, J. M. Carroll, and M. B. Rosson. Evaluating computer-supported cooperative work: models and frameworks. In CSCW '04, pages 112--121. ACM Press, 2004. Google ScholarDigital Library
- J. Nielsen. How to conduct a heuristic evaluation. http://www.useit.com/papers/heuristic/heuristic_evaluation.html.Google Scholar
- J. Nielsen. Finding usability problems through heuristic evaluation. In Proc. CHI '92, pages 373--380, New York, NY, USA, 1992. ACM. Google ScholarDigital Library
- J. Nielsen. Usability Engineering. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1993. Google ScholarDigital Library
- J. Nielsen. Usability inspection methods. In CHI '94: Conference companion on Human factors in computing systems, pages 413--414, New York, NY, USA, 1994. ACM. Google ScholarDigital Library
- J. Nielsen and R. Molich. Heuristic evaluation of user interfaces. In CHI '90: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 249--256, New York, NY, USA, 1990. ACM. Google ScholarDigital Library
- D. A. Norman. Cognitive Engineering. Lawrence Erlbaum Associates, Hillsdale, NJ, 1986.Google Scholar
- D. A. Norman. Cognitive artifacts. Designing interaction: Psychology at the human-computer interface, pages 17--38, 1991. Google ScholarDigital Library
- G. Olson and T. Moran. Commentary on" Damaged Merchandise?". Human-Computer Interaction, 13(3):263--323, 1998. Google ScholarDigital Library
- D. Pinelle, N. Wong, and T. Stach. Heuristic evaluation for games: usability principles for video game design. In Proc. CHI '08, pages 1453--1462, New York, NY, USA, 2008. ACM. Google ScholarDigital Library
- P. Rabardel and G. Bourmaud. From computer to instrument system: a developmental perspective. Interacting with Computers, 15(5):665--691, 2003. From Computer Artefact to Instrument for Mediated Activity. Part 1 Organizational Issues.Google ScholarCross Ref
- Y. Rogers. Ghosts in the network: distributed troubleshooting in a shared working environment. In CSCW '92: Proceedings of the 1992 ACM conference on Computer-supported cooperative work, pages 346--355, Toronto, ON, Canada, 1992. ACM. Google ScholarDigital Library
- M. B. Rosson and J. M. Carroll. Usability engineering: scenario-based development of human-computer interaction. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2002. Google ScholarDigital Library
- P. Sarbanes. Sarbanes-Oxley Act of 2002. In The Public Company Accounting Reform and Investor Protection Act. Washington DC: US Congress, 2002.Google Scholar
- B. Shneiderman. Designing the User Interface: Strategies for Effective Human-Computer Interaction. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA, 1997. Google ScholarDigital Library
- B. Shneiderman. Creating creativity: user interfaces for supporting innovation. ACM Trans. Comput.-Hum. Interact., 7(1):114--138, 2000. Google ScholarDigital Library
- A. Sutcliffe and B. Gault. Heuristic evaluation of virtual reality applications. Interacting with Computers, 16(4):831--849, 2004. Human Computer Interaction in Latin America.Google ScholarCross Ref
- D. Te'eni, J. Carey, and P. Zhang. Human Computer Interaction: developing effective organizational information systems. Wiley, 2007. Google ScholarDigital Library
- R. S. Thompson, E. M. Rantanen, W. Yurcik, and B. P. Bailey. Command line or pretty lines?: comparing textual and visual interfaces for intrusion detection. In CHI '07: Proceedings of the SIGCHI conference on Human factors in computing systems, page 1205, San Jose, CA, USA, 2007. ACM. Google ScholarDigital Library
- N. F. Velasquez and A. Durcikova. Sysadmins and the need for verification information. In CHiMiT '08: Proceedings of the 2nd ACM Symposium on Computer Human Interaction for Management of Information Technology, pages 1--8, New York, NY, USA, 2008. ACM. Google ScholarDigital Library
- N. F. Velasquez and S. P. Weisband. Work practices of system administrators: implications for tool design. In CHiMiT '08: Proceedings of the 2nd ACM Symposium on Computer Human Interaction for Management of Information Technology, pages 1--10, New York, NY, USA, 2008. ACM. Google ScholarDigital Library
- K. J. Vicente. HCI in the global knowledge-based economy: designing to support worker adaptation. ACM Trans. Comput.-Hum. Interact., 7(2):263--280, 2000. Google ScholarDigital Library
- K. Vredenburg, J.-Y. Mao, P. W. Smith, and T. Carey. A survey of user-centered design practice. In CHI '02: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 471--478, New York, NY, USA, 2002. ACM. Google ScholarDigital Library
- R. Werlinger, K. Hawkey, and K. Beznosov. An integrated view of human, organizational, and technological challenges of IT security management. Journal of Information Management & Computer Security, 17(1):4--19, 2009.Google ScholarCross Ref
- R. Werlinger, K. Hawkey, D. Botta, and K. Beznosov. Security practitioners in context: Their activities and interactions with other stakeholders within organizations. International Journal of Human-Computer Studies, 67(7):584--606, March 2009. Google ScholarDigital Library
- D. Zager. Collaboration as an activity coordinating with pseudo-collective objects. Computer Supported Cooperative Work (CSCW), 11(1):181--204, 2002. Google ScholarDigital Library
- J. Zhang, T. R. Johnson, V. L. Patel, D. L. Paige, and T. Kubose. Using usability heuristics to evaluate patient safety of medical devices. Journal of Biomedical Informatics, 36(1--2):23--30, 2003. Patient Safety. Google ScholarDigital Library
- A. T. Zhou, J. Blustein, and N. Zincir-Heywood. Improving intrusion detection systems through heuristic evaluation. In in IEEE Canadian Conf. on Electrical B. and Computer Engineering (CCECE), pages 1641--1644, 2004.Google ScholarCross Ref
Index Terms
- Heuristics for evaluating IT security management tools
Recommendations
Heuristics for evaluating IT security management tools
CHI EA '11: CHI '11 Extended Abstracts on Human Factors in Computing SystemsThe usability of IT security management (ITSM) tools is hard to evaluate by regular methods, making heuristic evaluation attractive. However, ITSM occurs within a complex and collaborative context that involves diverse stakeholders; this makes standard ...
Evaluating a Methodology to Establish Usability Heuristics
SCCC '12: Proceedings of the 2012 31st International Conference of the Chilean Computer Science SocietyAssessing usability in any software product may be a key factor for predicting its success or fail. Heuristic evaluation is the most commonly used usability evaluation method. It uses a set of recognized usability design principles (heuristics). Until ...
Detailed Usability Heuristics: A Breakdown of Usability Heuristics to Enhance Comprehension for Novice Evaluators
HCI International 2020 - Late Breaking Papers: User Experience Design and Case StudiesAbstractHeuristic evaluation (HE) is one of the most commonly used usability evaluation methods. In HE, 3–5 evaluators evaluate a certain system guided by a list of usability heuristics with the goal of detecting usability issues. Although HE is popular ...
Comments