Conducting a Usability Evaluation of Decentralized Identity Management Solutions

Despite of numerous state-sponsored efforts over the last twenty years to provide European citizens with high assurance electronic identities, today’s digital identity market is heavily dominated by single-sign-on solutions that are offered by big international corporations. Nevertheless, the market for digital identities is not saturated yet. Many use cases still wait for a suitable, e.g. interoperable, easy-to-use, widely adopted, secure, privacy friendly ID-solution. This is why market researchers still see a high potential in this sector [1, 2]. New approaches, initially based on Blockchain and distributed ledger technologies (DLT) have been getting a lot of attention in the past 3–4 years and are promising to disrupt the digital identity market [3]. Such approaches, often also called Decentralized Identity Management (DIdM) and Self-sovereign Identities (SSI), are often marketed as the future of digital identity management [4, 5]. In the following, we will use the term Decentralized Identity Management solutions (DIdM) for these approaches.

Numerous companies and projects (Sovrin, Jolocom, W3C Decentralized Identifier Working Group, Decentralized Identity Foundation, Hyperledger of the Linux Foundation etc.) are currently working on technologies that make it possible to use decentralized identities for trustworthy and privacy-friendly identification in digital interactions. The European Commission supports such approaches, for example through the European Self-Sovereign Identity Framework (ESSIF) as part of the European Blockchain Service Infrastructure (EBSI) [6] and the SSI eIDAS bridge [7]. Those actors see security and privacy as the main challenges in the currently available approaches to digital identities and promise to give the user the power to reclaim control over their own identity data in digital interactions [8, 9].

Nevertheless, experience shows that technical functionality even together with high levels of security and privacy protection are not sufficient for the diffusion of new information technologies [10]. Many technologies in the identity management sector that were previously regarded as disruptive, such as CardSpace, Uprove, and Attribute Based Credentials, have failed to find adoption by a significant share of the market [11, 12]. Hence, it has been argued that the consideration of multidisciplinary aspects, such as security, usability, and socioeconomics, is crucial for the success of a software product on the market [13]. However, it is a common issue that developers tend to focus on the former aspects while neglecting the latter [14].

We agree that there remains a growing need for identity management solutions to replace username/password and the solutions provided by the mighty GAFA platform-corporations (Google, Apple, Facebook, Amazon) — as has been countlessly discussed in information (security/privacy) science as well as in the public. New approaches entering the market for alternative identity management solutions still struggle with its multi-sided structure, leading to a “chicken or the egg” dilemma. Uptake with end-users and services providers and the sustainable as well as balanced trust relationship among the relevant stakeholders is a big challenge [15]. Therefore, we think that is important to critically assess the current promises, intentions and practices of DIdM solutions, in order to avoid mistakes that could, again, lead to the failure of a promising technology on the market. This is what we aim for with this paper. Our analysis is focused on the usability aspects of DIdM solutions, and studies them empirically with end users. Our research approach addresses the challenge of usability in DIdM solutions by conducting a user study that analyses and evaluates currently available DIdM solutions towards their practical applicability for end users. This paper presents the results of the usability tests with end-users that will be later used to build a user-friendly prototype and to give design guidelines to DIdM solution developers aiming to increase the adoption potential of their products. Other important aspects, such as the perspectives and requirements of service providers (relying parties) [16], will have to be kept aside for future work.

Our paper is structured in the following way. In section two we give an overview of the background and current state of Decentralized and Self-sovereign Identities. We briefly introduce the approach and terms, principles being followed, market overview and related research. Next, in section three, we present our user test of three DIdM solutions, describing methods, results, and analysis. Continuing, in section four we discuss the key results of the study and what we regard as its main outcomes. Lastly, section five presents the limitations and section six briefly concludes our paper.

Decentralized and Self-sovereign Identities (SSI) are currently being marketed as the future of digital identity management as opposed to traditional approaches that are often simply called “legacy systems” [17‐19]. The promise of these approaches is that they are able to empower users to take back control over their data [20, 21], and to overcome the dominance of the GAFA platforms [22, 23].

When it comes to new, alternative, decentralized approaches towards identities, the term Self-sovereign Identity has become more and more prominent. Although it is not always being used consistently, Mühle et al. [24] summarize the key properties of the concept as that a Self-sovereign identity management system would allow users to fully own and manage their identity without having to rely on a third party. The origin of the concept under this name can be traced back to 2016, when Allen published his so called “Ten Principles of Self-sovereign Identity” [20]. There, he also refers to the earlier proclaimed “Laws of identity” by Cameron [25], which illustrates that the basic approach is not entirely new. Following the taxonomy by Lesavre et al. [26], Self-sovereign Identity can be seen as a bottom-up approach, where no single entity acts as central authority that has control over identifier origination and/or credential issuance. Identifiers and credentials are solely managed by the users, without requiring any permissions. On the other side of the spectrum would be top-down approaches relying on central authorities as identity providers and federated approaches somewhere in between.

However, it is important to recognize that the reasoning provided for the DIdM approach and the SSI principles or laws are not founded on empirical studies of the requirements of users (and neither service providers). In addition, there are still some open questions on whether the users actually desire so much control and whether the solutions not only provide users with the theoretical opportunity to exercise this control through their technical architecture, but also empower them in practice and not confuse and overburden them. For users to be able to fully manage and own their identity without having to rely on a third party, they are required to somehow understand the concept and be assisted with usable tools that do not frustrate them. After all, typical users do not use identity management because it is such great fun, but rather to access services they want to use. Being in theoretical control of their identity could become a similar experience as the current total control users have over trackers and cookies when visiting a website. Having to manage those detailed settings manually through annoying dialogues might simply frustrate them. The lack of usability could then lead average users to simply use privacy unfriendly, but convenient solutions—a pattern we frequently observe.

One of the potential usability issues of Blockchain-based DIdM solutions is the fact that the private key that ensures the access to user’s personal data is in total responsibility of the user [27]. While this is often marketed as one of the most significant benefits of DLT-based and DIdM solutions, it also comes with significant challenges such as how to securely store and manage those keys to avoid irretrievable loss of key and connected accounts [28]. If such issues are not properly explained and handled, end users will have troubles understanding and using the new technology. This makes the solutions less attractive to average users and leads to lower levels of adoption. Especially, if it may seem that they require more and complex user involvement while not offering other benefits except for being more privacy friendly (something the average user cannot even personally assess as we are clearly in some kind of “market for lemons” here [29, 30]). Moreover, if such solutions are not widely supported by service providers and thus not integrated into a sufficient variety of services, their value for end users is even lower.

The issue of usability in privacy and security tools has already been subject of research efforts for quite some time [31, 32]. Nevertheless, there seem to be only few attempts to explicit fix user experience challenges for security tools and so also for DIdM. Following [33‐35], the major problems that go beyond the mere graphical user interface are:

This seems particularly important when it comes to a technology like DIdM that puts as much power into the hands of the individual user and builds on concepts such as public and private keys that are not trivial to the average online-shopper. We see this lack of consideration of the usability and mental models in current approaches to DIdM as an important potential weakness that needs to be studied empirically with explicit user involvement. Therefore, we want to address it through our work.

For our user study, we are dependent on publicly available and testable DIdM solutions. In their public presentations, proponents of DIdM give the impression, that the technology is ready to replace legacy IdM systems. On their website, for example, uPort writes of “easy-to-use data management and control to your business and customer” [36], Evernym of “The fastest, most efficient way for organizations to offer an SSI-enabled solution for their users” [37]. That of course raises high expectations regarding their solutions technology readiness and current practical applicability.

However, another consideration that needs to be made is that DIdM technology is still under heavy development and different approaches are currently being pursued. For example, those can be differentiated according to organizational structure, models for identifier and credential management, presentation disclosure, general system architecture design and the use of public registries. A systematic overview and discussion can be found at Lesavre et al. [26]. First standards are currently being finalized, e.g. by the W3C [38] and the DID [39], but the work is still ongoing. This makes interoperability between the different approaches challenging.

At this time, a significant number of companies and projects are working on decentralized identity solutions. In a thorough survey of market of Blockchain-based Identity Management solutions and technologies, Kuperberg [27] analyses 43 approaches with different levels of maturity and availability. He concludes that only a couple of these approaches could compete with established solutions when it comes to end-user convenience, though it has to be noted that his analysis has to remain on a rather high level due to the high number of solutions considered. What he misses in particular, is a clear and sustainable business model. Dunphy and Petitcolas [33] analyse IdM schemes of three Blockchain-based products (uPort, ShoCard, Sovrin) according to the Cameron’s “laws of identity” [25]. Regarding usability, they conclude that all of those projects have an “unclear usability and user understanding of […] (the) privacy implications.” None seems to actively address the issues in regard to fitting mental models and usability. All this apparently supports need for our research approach.

In our research, we are addressing the challenge of usability in DIdM solution by conducting a user study that analyses and evaluates DIdM solutions that are currently (beginning of 2020) available on the market towards their practical applicability and acceptance for end users. For our study, we identified 23 DIdM solutions. In April 2020, those identified solutions were on a sufficient level of maturity (and/or transparency in public communication) to provide enough information for an analysis that would determine whether they would be suitable for an end-user study. In order to identify which DIdM solutions would best fit a usability study with end users, we analyzed those 23 DIdM solutions by their differences in maturity, purpose and functionality. The suitability of solutions for the user test was evaluated according to the following set of requirements: DIdM as the core technology; minimal level of technology readiness (at least TRL 7); wide functionality (to be able to carry out at least 3 scenarios for user testing); availability of the wallet for both iOS and Android platforms; availability of a demo scenario or even real services to test the solution; interface language English and/or German. According to these conditions, three DIdM digital wallets qualified for then being evaluated in a systematic study including end-users in a usability test: Evernym ConnectMe, Jolocom SmartWallet, and uPort ID.

According to its advocates, the main benefit of SSI is to put the users in full control of their identities. This is supposed to help to protect their right of informational self-determination. However, with more control also comes the burden of more responsibility and more effort to manage and use these identities and credentials. To be able to manage them effectively, users need to form some sort of rough understanding of how the technology works (aka mental model). Though, our results show that the mental models of the users not necessarily align with those of the developers that are quite familiar with technologies like public key infrastructures end electronic signatures. Users quite often form a different understanding that is shaped by the traditional, hierarchical solutions they are currently using and therefore experience problems when trying to use and manage the credentials in a decentralized architecture. This is especially apparent when it comes beyond the simple use case of issuing and verifying credentials. Important aspects of the identity lifecycle like backup and recovery as well as deleting credentials or whole accounts, constitute huge challenges for the users of the DIdM solutions in our study.

These difficulties are further emphasized by the fact that the basic usability of current DIdM solutions leaves a lot to be desired. This leads to further frustration of the users. Another problem for the approach is that the development of the available solutions is often not as advanced as it is being advertised by their advocates. Essential features are often missing which can be observed by the fact that out of the 23 solutions we examined, only three could offer all the features that we required for our study. And even those three resembled more a work in progress than a mature market ready solution. As just one example, one wallet application (Evernym ConnectMe) in the Fall of 2020 completely removed the backup functionality. It had been available during the user tests, but an update of the application removed the function. Such a gap between promises and actual performance that can be delivered at this point could lead to exaggerated expectations that can only be disappointed if one wants to implement the immature technology right now. This might sustainably damage the reputation of DIdM solutions. The danger is that this could also be regarded as another example that privacy friendly solutions just do not work in practice.

Moreover, to be successful on the market, DIdM faces the same challenge as all other competing and often much more mature identity management solutions. It has to attract a high amount of users and relying parties to benefit from network effects in a two sided market [15]. To achieve this, the perceived benefit by users and relying parties or relative advantage of the DIdM solutions has to be higher than the competition. The main question will be if — in the eyes of the end users and service providers — the perceived benefit of more user control and more privacy will outweigh the drawbacks such as increased effort to manage the credentials (potentially more annoying dialogues to answer), higher responsibility e.g. to secure the device that is used to control the identity, more complicated backup in case of lost credentials, and particularly at the current state, poor usability and lack of maturity. An empirical user study on web identity management raises some doubts in this regard [43]. Their results show that users do not value control over their identity data as much as many proponents of DIdM apparently expect. Therefore, we believe that it is essential for developers of DIdM to address the current drawbacks we pointed out in a multidisciplinary fashion to improve the likeness of their success on the market.

Our empirical user study and the derived analysis have undoubtedly some limitations, particularly regarding the sample of end users that participated in the test, the testing setup and the development state of the digital products that were tested.

A sample of 18 participants certainly cannot be regarded as representative of the general population. Most of the test participants were young people around 30 years. In most of the cases they reported themselves as being tech-savvy and could speak a high level of English while living in a non-native English-speaking country, which points to a higher level of education. However, while this is certainly biased sample, we can reasonably assume the results of the user tests could have been even more negative for the case that a broader sample of participants would have been available for us. An example would be end-users who are not as confident with smart phones, scanning QR-Codes, and other relatively new technology.

Another fact that needs to be considered is that the participants tested the DIdM solutions at home having a good internet connection for their smartphone (and desktop computers if used as well). Thus, at least connectivity-wise the whole process (e.g. obtaining credentials) ran smoothly for most of them. However, even under such perfect conditions there were cases when the connection between the digital wallet and the demo website was broken for some time and there was no way of getting back seamlessly to the process. Some of the tested wallets did not offer any solution for such cases and users had to start the whole process from the beginning. It would be interesting to learn about how DIdM solutions relying on mobile smartphone wallets perform in practice when there is actually still a significant number of situations when smartphone connectivity is limited (Sign on at a desktop computer with no WiFi available for the smartphone and no high speed mobile internet connection).

Also, the evaluation of wallets was conducted at a particular time (April to June 2020) and only the three selected DIdM digital wallets had the level of maturity that was necessary for our user tests. However, even at that time these products were constantly changing significant aspects of their functionality (e.g. backups), one became temporarily unusable.

Finally, we could test the available digital wallets only with demo scenarios provided by the solutions themselves. The use cases were chosen by solutions, thus might be selective to work particularly well, and this was of course not a productive environment. Still, even in this optimized environment, the issues were apparent.

After conducting an initial analysis of 23 Blockchain-based DIdM solutions and performing 18 user tests with three of the more advanced applications, the current usability problem of DIdM and SSI solutions can be defined as significant. Principally, we found the overall issue that the new concept of decentralized identity while apparently seeming self-evident to its developers is not explained well enough to the end users, which leads to substantial problems that encumber the practical use and purpose of the technology.

In addition, the major importance of easy-to-use functionalities to backup and recover the account as fundamental step in the identity lifecycle does not seem to be understood by the developers. Its importance is not prominently highlighted in the applications — maybe as the functions currently are too complex for the average user and their practicality is debatable.

We want to conclude by highlighting the concern that even though such solutions are marketed as ready to be practically used, their usability and current state of the technology stack might deprive end users of experiencing the entire range of claimed privacy and security benefits. DIdM solutions that exist nowadays need to provide a solidified explanatory basis and carefully guide the user with a good user experience, for example through the interface. This requires an explanation that is beyond providing basic instructions on how to use certain functions but also providing clarification of why certain functions need to be carried out in one way, while solutions that are more traditional and familiar to users have been offering similar functions in another way. To sum up the results of our study, to our knowledge the existing market does not yet offer Blockchain-based DIdM solutions with usability mature enough to be accepted and securely used by end users.