Collaborative Work Enabled by Immersive Environments

Digitalization has a great impact on the way we work together [78]. The digital transformation facilitates new methods for remote collaboration while shaping a new understanding of working together [95]. To process collaborative work within Collaborative Virtual Environments, the involved collaborators and their work environment containing the required tools and artifacts need to be digitalized. The virtualization of collaborative tasks greatly benefits from the ongoing “Industry 4.0” movement involving Cyber-Physical Systems (CPS), Internet of Things (IoT) and Internet of Services (IoS) technologies which are relying on data collection and digitalization of the involved physical artifacts. The digitalization of the artifacts goes to the extent of Digital Twins (DTs) being continuously synchronized real-time representations of the physical objects within the Virtual Environment (VE) [74].

The other way around, collaborative networks are considered a core driver [9] and remote work an accelerator [23, 25, 53] for the digital transformation. The “Industry 4.0” concepts strongly rely on new organizational forms, mechanisms and processes with a collaborative nature in the main dimensions (i.e. vertical integration, horizontal integration, through-engineering, acceleration of manufacturing, digitalization, and new business models) in which collaboration related issues could be identified [9].

From the social perspective, the demand for more flexible work in terms of work location and work time is increasing and the lockdowns during the Covid-19 pandemic showed that remote work is possible with today’s technology in many areas [25]. As a result, the pressure on organizations is increasing to adapt and implement new ways of working [54]. Current research concludes that remote work will be accepted as a common way of working¹ complementing the ongoing trends from tele-medicine and tele-learning. As a side effect, the implementation of remote work will accelerate the digital transformation [40].

The Covid-19 pandemic plays a big role in the transformation of work and affects how collaborative work environments should be developed to meet the changing requirements. Recent research shows that Covid-19 serves as an “accelerator”, “catalyst” or “booster” for the ongoing transformation of work [23, 25, 53]. At the beginning of 2020, governments imposed lockdowns forcing public and private organizations to restructure their work processes and ensure safe work for their employees. Edelmann et al. [25] have researched how Covid-19 has affected work in the public sector since 2020. They observed a transition from traditional work in the offices to remote work in home offices during lockdowns and concluded a rapid and far-reaching change of work in organizations from all domains. To ensure safety, the workplace structure should change and have less populated work environments [23]. The pandemic has opened opportunities for changing traditional work to remote work, which became an important measure to combat Covid-19 while ensuring service delivery². For instance, the “boost” of remote work was observed in Austria: while remote office was used in 75% of the companies by a very privileged group of employees, after the onset of Covid-19 regulations in 80% of the companies most of the employees worked remotely, and 85% were utilizing tools for remote collaboration [48]. Before the pandemic, remote work from home was equated with “being on holiday” but in the situation during the Covid-19 lockdowns, the employee performance was perceived as equivalent to being in the office [25]. Remote work brings benefits of flexibility, greater concentration, better work-life balance, and a perceived productivity increase [25, 48]. On the other hand, it also brings disadvantages for remote collaborators like long online meetings, information overload, and difficulties with team coordination. Nevertheless, a large majority of employees wish to continue working remotely more extensively in the future [25, 92] which emphasizes the importance of the digital transformation of work. In fact, digital transformation requires more than just the application of digital technologies. The success of the digital transformation of work also highly depends on the adaptability of people [23]. Also, a change in the composition of required skills of the collaborators, like independent working and self-management for the employees, as well as providing respective learning opportunities and basic structures for remote collaboration can be perceived [25].

CVEs belong to the family of immersive technologies. By immersive technologies we refer to Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) technologies, also grouped into a generic term, eXtended Reality (xR) [88].

Milgram et al. illustrate the different concepts for immersive technologies with a simple diagram called the reality-virtuality continuum [71]. Figure 1 locates VR, AR, and MR within this continuum. Between the two extremes, the real and virtual environments, there are infinite combinations that are grouped under the umbrella term MR. Particular forms of MR are thus AR and Augmented Virtuality (AV). AR is defined as a computer-assisted augmentation of reality perception by superimposing virtual objects or additional information. AV, on the other hand, allows the developer to enrich the artificial world with real information. According to the continuum, VR is defined by its fully simulated environment [37]. Sherman and Craig [97] give a definition for VR as: “a medium composed of interactive computer simulations that sense the participant’s position and actions and replace or augment the feedback to one or more senses, giving the feeling of being mentally immersed or present in the simulation”. The essential properties of VR are immersion, interaction, imagination, presence, and intelligence [37].

VR systems, designed to provide a sense of presence, consist of a number of hardware devices and software. The hardware typically consists of visual and audio output, a tracking system, powerful graphics computers, and in some cases, may have haptic feedback. The visual output comprises three-dimensional screens that can be attached directly to the user’s head (called Head Mounted Displays (HMDs) or VR headsets, see Fig. 2) or a stationary screen such as a projection-based Powerwall or a Cave Automatic Virtual Environment (CAVE) (see Fig. 8). AR systems use mainly so-called see-through headsets (see Fig. 2 on the right) or hand-held devices (smartphones, tablets). The required software is based on VR-capable graphics engines or scene graph systems to execute the applications. This task could also be performed by a VR authoring system, which has the additional task of building the scene and implementing the application logic. Depending on the use case, 3D modeling tools can also be applied to create the 3D content. To realize CVEs the VR software needs a special collaboration module to enable communication and synchronize the user interaction with the VE. In this way, the VEs can be experienced by more than one user and collaboration between the users can take place. At this moment, we are talking about collaborative VEs or CVEs.

Collaborative Environments (CE) occur in many varieties depending on the type of collaboration itself and depending on the function it shall fulfill. For instance, the collaboration can take place at different times or involve different locations. CEs can enable various collaborative tasks like group authoring or virtual meetings. The matrix in Fig. 3 shows the dimensions of collaborative work and is an extension of the groupware matrix by Johansen [46].

The matrix of collaborative work classifies group work into six categories along the dimensions of space, time, and xR (see Fig. 1). In the dimension of space, a co-located collaboration takes place at the same location for all collaborators, while in remote collaboration the collaborators can participate from remote locations joining a shared environment. Along the time dimension, collaborative work can be conducted synchronously, when the participants are working at the same time, or asynchronously if they work at different times, like in globally distributed teams. Involving the xR dimension, the collaboration takes place within virtual environments, referred to as CVEs. In the xR dimension, symmetric collaboration describes a collaboration within the same reality-virtuality level, for instance in VR. On the contrary, in asymmetric collaboration the CVE representation can vary within the MR continuum between the collaborators. For instance, a part of the participants can be inside the AR representation of the CVE, while another part can participate within the VR one.

CVEs enable non co-located persons to meet in a virtual environment. They come in many varieties, from a simple chat room, an online game, to a virtual conference accessed through immersive hardware systems. This section focuses on the key aspects of CVE software, like connectivity, communication architecture, and interaction and collaboration paradigms. Furthermore, in Subsect. 4.2 hardware setups of possible projection systems are presented.

Connectivity is key to being able to connect applications over the internet. First, the applications have to be visible to each other to be able to meet up. This can be achieved in a local network by defining a system as main server. When connecting over the internet, it is in general not possible to address the systems via IP, as they are behind a Network Address Translation (NAT). In this case, it is necessary to have a match making server that can be reached from the internet. Depending on the communication architecture, the match making server can also have the role of a TURN server or even application server (TURN stays for Traversal Using Relays around NAT). A TURN server helps two peers to connect, the server then routes any data streams between the peers. An application server is required when the application uses a server-client based communication, where the server manages the application state. This is typical for online games where the server manages the application state like players, their progress, and the state of the environment.

The communication architecture is based on Representational State Transfer (REST), Transmission Control Protocol (TCP) and User Datagram Protocol (UDP) data streams. REST is typical for communicating control messages with servers, for example for match making or session management. TCP data streams are used for important data like model data transfers, user interaction, or application events. This allows to keep track of such data and detect inconsistent states between collaborators. UDP data streams are useful when transmitting audio and video streams and maybe tracking transformations. This kind of data stream is robust, as dropping data packets will not result in corrupting the application or significantly impact the user experience, at least if there is not too much loss due to a bad connection.

The available collaboration paradigms depend on the connectivity and communication architecture. The most basic setting is to display user avatars and a chat window to communicate. This results in a very small network communication data volume, as only the transformation matrices of the users have to be synchronized between application instances. Instead of a chat, it is also possible to start a conferencing tool like Zoom or Teams, which will increase the transmitted data volume. The next step to ease the communication is to enable voice over IP, this is more challenging on a technical level as the application needs to access microphone devices and stream the data to other nodes as well as play the voice streams locally of collaborating users. Even more challenging is synchronizing the scene graph, the simpler version is to just synchronize the visibility and transformation of 3D assets, but each asset is already shipped with each copy of the application.

It is also possible to synchronize any 3D content, for example generic triangle meshes, but this requires a complex state management of the VE, per frame change lists, change filtering, serialization and deserialization, and much more [68]. Pečiva [81] presents several strategies for the replication and synchronization of VE for remote collaboration. Furthermore, the work introduces the Active Transaction (Fig. 4) as a more efficient approach for synchronous remote CVEs.

This is the basis for Collaborative Virtual Environment (CVE) applications, where users may present and discuss any 3D data (e.g., CAD models) within a shared virtual environment.

The effective implementation and efficiency of CVEs are intimately linked to the effects that virtual immersion and associated technologies have on users, and therefore, by extension, to their ability to accept them. Indeed, virtual immersion may place users in different psycho-physiological and behavioral states than those observed in the real world. Numerous issues are to be considered when developing CVEs and here we present three major ones of those (Fig. 5).

Cybersickness. Among the recurrent issues of immersive technologies, cybersickness is undoubtedly one of the most important but also among the most studied. It is indeed known that virtual movements, undergone or carried out by a user, cause a characteristic malaise, with symptoms close to those of motion sickness: nausea, pallor, stomach ache, fatigue, etc., which can even lead to vomiting. If the phenomenon has been abundantly described in the literature [10, 49, 108], its explanation is still subject to numerous studies because of its high complexity. Nevertheless, we can summarize here the main theories associated with the appearance of cybersickness, the factors that can promote it, and the means currently developed to mitigate its effects.

The main theory commonly accepted to explain the appearance of cybersickness is the sensory conflict, introduced in 1975 by Reason and Brand [89]. This theory states that the human body uses the visual, vestibular and proprioceptive systems to orient itself in space, and that a conflict between the information received by these systems leads to sickness. In virtual immersion, it is very frequent that a virtual movement stimulates the visual system but not the vestibular one because the user does not move physically or only slightly. A conflict then arises between the two systems, causing cybersickness. Another theory, also very widespread, is the ecological theory, proposed by Riccio and Stoffregen in 1991, according to which sickness comes from prolonged periods of postural instability [91], which may themselves be triggered by a dysfunction of the body’s balance mechanism in the presence of visual stimuli [103]. This theory has highlighted the fact that a significant amplification of the postural sway precedes the onset of sickness [102], and several studies in virtual reality have helped to better characterize the features of postural sway in order to anticipate the occurrence of cybersickness (e.g., [12]). A third theory, called the evolutionary theory and developed by Treisman in 1977, proposes to assimilate motion stimuli to intoxication, triggering corresponding reactions by the body [109]. However, this theory is not widely accepted, as several studies have pointed out inconsistencies [38, 77]. The last theory, less known, is the rest frame theory, introduced in 1998 by Prothero [86]. This theory, considered a refinement of the sensory conflict theory [85], stipulates that a person uses stationary references in space, and that when an appropriate reference cannot be chosen, for example, in the presence of motion cues, cybersickness occurs.

The levels of cybersickness experienced are highly dependent on several factors. On the one hand, as each person is different, parameters related to each person can influence how this phenomenon may be experienced. Past studies have shown, for example, that people suffering from balance organ disorders in their inner ear are insensitive to motion sickness [43, 120]. Similarly, females seem to be more susceptible than males to cybersickness, due to differences in hormonal levels [33], in the field of view [57], but also in the way sickness symptoms are reported [5]. Age is also considered as a parameter that can influence sickness [20], but opinions are more divergent, as sensitivity can also be affected by life experience, particularly that in video games; a person accustomed to video games is able to develop abilities to move quickly and accurately in a virtual environment [99], which desensitizes from cybersickness [31, 98]. Ethnic, genetic, and hereditary factors may also be involved in susceptibility to cybersickness [51, 52]. On the other hand, parameters related to the technology itself can influence the levels of cybersickness. Among these, we can cite the type of immersive device used (e.g., Head Mounted Display (HMD) or Cave Automatic Virtual Environment (CAVE)-type system) [66], the field of view of immersive devices [26], which is usually smaller for most devices on the market than the human one, a high latency of the system [83], low frame rates [57], poorly adjusted interaction parameters [116], unsuitable interaction interfaces [70], or long-time immersion [11].

Many works have tried and continue to try to reduce the effects of cybersickness. One way is to improve existing virtual displacement techniques, by adapting navigation parameters according to the environment in the immediate vicinity [13] or by taking care not to exceed certain speed and acceleration thresholds [105, 124], but also by adapting the field of view and adding blur in the peripheral vision according to the motion [14, 104]. Another method is to develop new techniques of virtual displacement, including teleportation-based [72], motion-based and room-scaled-based such as redirected walking [28], or controller-based techniques [50, 115]. Cybersickness can also be minimized by using motion platforms such as omnidirectional treadmills [61], which however may require space and be costly. Another solution can be to use a guiding avatar [60], representing the user in a third person perspective, as in many video games and more recently in virtual world applications (e.g., Virbela, Horizon Worlds or MeetinVR), though further studies are necessary to confirm its efficiency. More recently, work has been done to recenter interaction in virtual environments on the user, or even to individualize the interaction, with the assumption that on the one hand the level of presence in the virtual environments will increase and on the other hand the levels of cybersickness will decrease. Thus, it is possible for example to limit cybersickness by adapting virtual accelerations according to the user’s physiological state [82]. Similarly, with the massive development of artificial intelligence tools, one promising perspective is to predict the onset and thus minimize the levels of cybersickness based on the users’ profile [117] or behaviors and activities [44].

To summarize, cybersickness is a recurrent phenomenon to be carefully considered to prevent users from rejecting immersive technologies. It is doubtful that cybersickness will be one day eliminated. However, one should make sure to always minimize it. All parameters, including human and technological ones, should be reviewed to provide optimized immersive experiences. Minimizing latencies, carefully designing immersive scenarios, and taking users’ peculiarities into consideration in the design of interactions, are among the ways to succeed.

Therefore, it seems crucial to optimize the usability of an xR interaction system for its better acceptance. According to the ISO 9241-11:2018 standard, the usability of a system is defined by its ability to allow users to perform tasks in an efficient, effective and satisfactory manner [45]. Thus, in xR, the question is how to interact with a virtual environment fulfilling these criteria?

Intuitively, it can be thought that the interaction allowing the greatest usability is natural interaction. Indeed, this type of interaction requires devices that are generally non-intrusive for the user, such as a microphone or a camera, and exploits skills acquired at an early age, such as walking, talking and gesticulating. However, the integration of such interactions in xR cannot be achieved without constraints. In VR, the particularity, as seen above, is to be immersed in a virtual environment and for that, it is necessary to use devices such as HMD or screens of more or less size. However, these devices and the systems necessary to operate them impose physical constraints in an inherent way, e.g., limited tracking space, limited physical movement space due to the presence of screens. Thus, to navigate in a virtual environment for example, if it is larger than the physical space, navigation techniques and devices must be employed, as natural walking becomes very quickly constrained. We have seen above several categories of existing navigation techniques. A technique that has attracted the VR community’s attention for a few years is redirected walking [28]. Its principle is to make users believe that they are walking in a straight line in the virtual environment while physically they are walking in a curve so that they remain in a constrained physical space. If the undeniable advantage of this technique is to allow users to walk in a natural way, it nevertheless requires a fine-tuning of the redirection parameters which, if improperly done, can make the experience unsatisfactory and thus the system unusable [56]. Gesture and speech interaction suffers from problems related to gesture and speech recognition, which assumes on the one hand that the system has a pre-recorded or learned knowledge base and on the other hand an accuracy and recognition rate acceptable to the user. On this last point, recent works show increasingly high recognition rates (e.g., [122]) but which do not yet reach 100% or do not allow an optimal usability [1]. This, in turn, can affect cognitive load [30]. Therefore, even though natural interaction seems at first glance seducing, there are still avenues for research to get natural interaction effective, efficient and satisfactory to realize tasks in xR.

The same thought goes for computer-based interaction, involving interaction devices, such as controllers or joysticks, but also interaction techniques. Several studies investigated the usability of immersive interactions (e.g., [55, 69]) and how cognitive load can affect user experience (e.g., [123]). Overall, the simpler the interaction, the better [69]. Practically, that means also that when operating an immersive application, the fewer explanations are needed to use the system, the better. A last point is to ensure that interaction parameters (i.e., navigation speed, acceleration) are well tuned to maximize chances of usability. For instance, George et al. proposed a navigation method in virtual environments consisting in scrolling on a smartphone as on webpages [32]. Although smartphones are commonly used, the navigation parameters were not tuned properly, preventing users from navigating smoothly and therefore degrading user experience.

A last aspect to consider in interaction relates to multi-modality. Here, multi-modality consists in soliciting other senses than vision. It is well known now that integrating, for instance tactile or haptic feedback is of major importance for enhanced user experience [21, 119, 121]. A huge literature exists on prototypes of devices allowing tactile and haptic feedback (e.g., [16, 58, 79]). There have been companies selling haptic gloves and arms for more than 25 years, however they were released at a time when VR was not affordable and still in its infancy. Recently, thanks to miniaturization, lower component costs, and the increasingly widespread of xR technologies, several startups and companies have released new various and more affordable haptic systems, including gloves and suits, to provide tactile, haptic, even thermohaptic feedback, such as Manus³, HaptX⁴, WEART⁵, or Actronika⁶. Aside from haptic feedback, stimulating the vestibular system is another way to improve user experience and contribute in reducing cybersickness effects. Sensory stimulation can be made through motion systems, as massively performed in driving simulation, where multiple degree-of-freedom platforms are used to render realistic vehicle motion with appropriate motion cueing algorithms [49]. Such systems can be used in various use cases, such as training in hazardous situations [21]. Other sensory cues, such as auditory or olfactory, can be integrated to further immerse users.

To summarize, the acceptance of xR technologies is dependent on how well they fit users’ needs and experiences. As efficiency in work is always sought, immersive applications should be designed in a way to ensure maximal usability. The simpler, the better. Integrating multisensory feedback provides added value to user experience, however it is not a question of combining a maximum of technologies if the use case considered does not require it, at the risk of producing the opposite effect of that sought. Let us imagine for example an immersive training situation in maintenance involving the closing of valves. If the main objective is to learn the maintenance procedure rather than the way to close valves, there is no need to ask operators to do the gesture of closing accurately, nor to include haptic feedback; just virtually touching the valve and pressing a button on the VR controller to trigger automatic closing is sufficient. Therefore, when designing an immersive application, it is of primary importance to build its specifications properly to derive the right choice of technologies.

Perception of Virtual Environments. Another aspect to be considered is to ensure that end-users perceive virtual environments correctly. Typically, past work has largely reported misperception of distances and scales in VR, with observations revealing underestimations of distances up to 50% compared to reality [90], and virtual objects appearing smaller than in reality [101]. Such effects may disturb end-users in tasks, such as product design reviews or navigation in wide environments, in which size or distance checks may be required. In fact, such inconvenience may lead to a loss of presence in the virtual environment [41]. Reasons are not yet fully understood and are still the object of intense research. Though, it is known that a mix of factors may affect perception. For instance, the weight of a head-mounted display may influence distance perception above three meters [19]. Graphics quality is often cited as an impacting factor, including for example avatar realism [76], though some research does not consider it as significant [107]. More important is the way virtual cameras are set with respect to users. The inter-pupillary distance for instance should be tuned accurately in immersive systems (both physically of the devices and by software) to each user [118], which may not always be easy, as some head-mounted displays for example do not offer the possibility to tune this distance outside a pre-defined range. Lenses present in HMDs may also bring distortion of the displayed images. Other parameters from the display can also be considered, such as resolution, contrast, brightness or refresh rates [106]. Distances of users and virtual objects to the screen [7, 64], or the cognitive profiles of end-users [6] are other factors known to impact distance perception.

To remedy misperception in virtual environments, past studies revealed that performing actions, such as navigating in the virtual environment or manipulating virtual objects, may positively affect distance perception [113], within low-to-medium-range distances (usually less than 100 m) [39]. Furthermore, a correct setting of immersive devices and VR software is necessary. Simple checks can be performed to verify that the parameters of immersive systems are correctly set up [96], including checking the inter-pupillary distance to fit each user, verifying the users’ eye location in both horizontal and vertical directions, checking the proportions and sizes of virtual objects in screen distance, or verifying that size changes correctly with varying distance.

One of the open issues in the research of immersive collaborative environments is to enable switching between different collaboration modes and representations, for instance the change from co-local synchronous collaboration in AR to remote asynchronous collaboration in VR [27]. To enable more flexibility in collaborative work, CVEs should enable a transition between the different collaboration modes within the space, time and xR dimensions (Fig. 3).

Space Dimension. In the space dimension, the CVE should support transitions between co-local and remote collaboration. At the current state, collaborative work takes place either remotely, like in video conferencing, or co-locally, like in personal meetings. In xR CVEs are possible which will allow the participants to transit seamlessly between these two modes [27] allowing more mobility during collaboration while keeping the natural communication form.

For instance, collaborators who are running late for a meeting can virtually join while being on their way to the meeting location. With MR technology the virtual avatars of the remote collaborators are projected into the meeting environment similar as in Fig. 2. The representation of the users within the 3D meeting environment is enabled by immersive technologies. As a virtual representation of the remote participants, 3D avatars enable a natural communication between the present and distant collaborators, including verbal speech and non-verbal cues like gestures, gaze, and facial expressions. Compared to video calls, immersive CVEs improve collaboration by including spatial information. For instance, immersive CVEs allow the communication of deictic gestures and gaze directions, which is not possible in traditional video calls but play an important role during face-to-face communication. As soon as the remote collaborator arrives at the meeting location, the CVE system is responsible for switching the collaboration mode from remote to co-located. In particular, the avatars and digital speech transmission must be disabled for all co-located participants and enabled for the remote participants.

Time Dimension. In the dimension of time, the recommendation is to provide possibilities to switch between synchronous and asynchronous collaboration modes [27]. Synchronous collaboration in immersive CVEs is a well-studied subject [3, 27] and many companies already offer services and software products [112] which can be used within distributed teams as a shared CVE. Unfortunately, that is not the case for asynchronous collaboration in immersive CVEs [3, 18, 27, 42]. Reviews on MR collaboration highlight the benefits asynchronous collaboration methods can bring to different application areas and emphasize that future CVEs should offer possibilities for both, synchronous as well as asynchronous collaboration [3, 27].

Before exploring the transition between synchronous and asynchronous collaboration, more research is needed for understanding and defining concepts for asynchronous work in immersive environments.

In asynchronous collaboration, the participants do not have to be present at the same time within the collaborative environment to be able to conduct collaborative work together. In other words, the collaboration takes place over an extended time duration [42]. The collaborators can progress on their tasks independently within the CVE which will track and merge their changes to ensure a consistent state of the work for all participants.

The majority of the works show concepts for leaving annotations in the virtual space for information exchange between the collaborators, but asynchronous collaboration in immersive CVEs can go far beyond the scribbling of static virtual content and its consumption at a later time [42]. Unlike with static annotations, dynamic behaviors and actions can be preserved within the collaborative space and be presented on demand for later consumption. In fact, the work on a shared task can be visualized within the working space by 3D avatars of the previously working collaborators without them having to participate at the same time.

A concept for conducting manual work asynchronously in VR is presented in [67]. The collaboration is considered to take place between an expert on a specific assembly task and a trainee who is being asynchronously instructed by the expert, as depicted in Fig. 6. In the presented concept, the movements and actions of the collaborators are recorded by the VR system while they are working on the assembly task. To instruct the trainee, the expert can capture his actions within the 3D space and send the record file to his trainee. The trainee can proceed with the training instructions at any time. Therefore, the received recording can be replayed within the CVE, allowing the trainee to see a ghost of his trainer showing the assembly steps including the ghosts of the involved assembly parts. The ghost visualization has been chosen to distinguish non-present collaborators and objects from present ones. During the replay of the ghost trainer, the trainee can work with the present objects, imitating the assembly process. Additionally, the actions of the trainee can be recorded by the CVE and sent back to the trainer for evaluation and if needed corrections.

This concept, including the immersive capture and replay of the collaborators’ actions and the interaction with their results, enables collaborative work on spatial tasks even if participants are not available at the same time. Besides training scenarios, the concept can be applied to the documentation of cooperative work, team performance reviews, remote maintenance, co-design and many more.

xR Dimension. Regarding the xR dimension, CVEs should enable switching between different representations during the collaboration. For instance, the collaboration can begin in AR for the co-present collaborators and continue in VR when they switch to remote collaboration. Despite the change, the collaborative experience should remain consistent for all participants. Figure 7 shows first tests on asynchronous remote collaboration between a VR and an AR user [67]. Therefore, the CVE as well as the exchanged data between the collaborators must be compatible regardless of the xR device (whether VR or AR).

Depending on the xR device, the visualization of the remote collaborators and the tracking of the local user may vary. Regarding the representation of the user’s hands, most VR devices allow the usage of hand controllers for input, while the majority of AR devices rely on hand tracking. Therefore, in VR only the movements and actions of the controllers are tracked and need to be mapped to a virtual hand representation (Fig. 7 left). Meanwhile, with hand tracking in AR the hands are recognized by the AR system tracking the hand joints which can be used to animate the virtual hands (Fig. 7 right). As a result, the CVE has to support several hand representation and tracking techniques and handle the mapping of the different hand models.

Furthermore, VR requires virtual objects to interact with, while AR allows the interaction with physical objects. Within the remote maintenance scenario, a remote expert could explain maintenance steps on a virtual representation of the physical objects to an on-site worker trying to reproduce the steps on the physical objects. This scenario requires the localization of the physical objects within the on-site working environment. Once localized, the physical objects can be overlaid with virtual holograms of the remote expert. On the expert side, a virtual representation of the physical objects is needed to create the guiding instructions, which can be obtained for instance from CAD data.

To measure the usability of a system, three metrics are proposed in the ISO 9241-210:2010 standard: efficiency, effectivity, and satisfaction. For each of the metrics, different tools can be utilized to elicit quantitative or qualitative measurements. For instance, effectiveness can be measured by analyzing the completeness or correctness of a task performed within the CVE, efficiency by measuring the required time to complete the task and the satisfaction via user feedback in a questionnaire (i.e., System Usability Scale (SUS) questionnaire).

Quantitative data is usually measured via sensor feedback, while qualitative data can be sampled from user questionnaires and is therefore hard to compare between the study subjects. Nevertheless, qualitative results can be used to clarify and explain the quantitative results.

Physiological sensor measurements can be used to evaluate the user experience of CVEs using non-invasive data collection methods such as eye tracking, electrodermal activity, or heart rate [47, 62]. Physiological signals provide valid and often unique data that is of particular interest to the user experience. Physiological measurements can provide deep insights into effects such as immersion and presence. They can quantitatively and objectively measure users’ subjective experience while interacting with the virtual world [47]. Unlike self-assessment or low-scoring questionnaires used during or after immersive activity, physiological measures can be used to collect user data at the precise moment when users are engaged, immersed, or in flow without interrupting the user experience. For the studies with collaborative environments, a large sample (more than 30 users per test group) should be used because the fidelity, and therefore the complexity, of physiological measurements is much higher. To measure the study variables, methods with different levels of fidelity should be used. Such methods include tracking variables to map different motivational states (between traditional and immersive collaboration sessions), embedded surveys to measure usability, physiological signals such as heart rate, electrodermal activity to measure presence using near-infrared spectroscopy (NIRS), and other non-invasive methods. Results can be presented as inferential statistics and calculated using quantitative methods such as multivariate analysis of variance (MANOVA). In this way, it is possible to verify whether there are statistically significant effects on specific variables compared to other collaborative instruments.

In the business application area, CVEs are mostly utilized to have face-to-face meetings despite working from remote locations as depicted in Fig. 2. This is enabled by immersive CVEs providing a 3D meeting environment as well as personalized 3D avatars representing the participants. As in physical co-located meetings, communication cues like facial expressions, deictic gestures, and gaze directions are important to have an efficient discussion and have to be visualized during the collaboration. Furthermore, the CVE should contain common meeting tools, such as whiteboards, post-its, and markers and allow intuitive interactions with the virtual objects as discussed in Subsect. 3.1.

For business meetings, the support of other multimedia sources inside the CVE is important. Usually, meetings involve the usage of digital documents, websites, 2D assets like images and videos as well as 3D assets like 3D models. Another significant requirement in this area is the support of massive collaboration. CVE meetings can involve small teams with up to ten participants, but also hundreds of participants, for instance during virtual exhibitions and conferences.

Within the business area, CVEs can furthermore be used for collaborative data analytics and decision-making, utilizing the third dimension immersive interaction to get a better overview of the massive data sets.

Finally, for marketing and procurement purposes, CVEs enable to approach customers remotely and directly in their homes and institutions. Virtual previews of physical products can be customized and experienced in an immersive space, allowing a better idea of how the product will finally look like and behave. With the use of AR the virtual products can even be placed inside the own homes⁷ or on our own bodies⁸.

An application domain in which collaborative work is fully integrated is engineering, in particular Product Lifecycle Management (PLM), and more recently in the construction field with the Building Information Modeling (BIM) process. The PLM cycle, specific to each product or service developed, involves several stages, among which design, manufacturing, marketing, maintenance, and recycling. During these different stages, collaborative sessions can take place between collaborators according to the modalities explained in Subsect. 2.2. These collaborators can come from different fields but all work in parallel on the same virtual support, a digital mock-up (DMU), integrating the data from all fields, and thus carry out concurrent engineering [111]. With BIM, it is even more challenging as mock-ups should continuously be updated, not only during construction and until the delivery of the buildings, but also during usage, to facilitate management for maintenance or update operations, with the consideration that a majority of stakeholders (e.g., some craftsmen) may have not adopted computer-supported tools [87].

Virtual and augmented reality has emerged as an indispensable technology that facilitates system development while reducing development time and costs, but also increasing user safety in hazardous situations⁹. The literature is rich in propositions of system development methods, collaborative or not, using virtual reality (e.g., [17, 29, 63]). Virtual reality allows, for example, the visualization at real scale of the digital mock-up and interaction in a natural way (e.g., turning around the mock-up, selecting components, simulating components assembly). In concurrent engineering, it is possible to imagine each collaborator using his/her own immersive visualization device (i.e., a CAVE-type system, an immersive headset, a powerwall), visualizing the representation specific to his/her field of the same digital mock-up, and exchanging on modifications to be made or the validation of the mock-up. Thus, for example, for the design of a vehicle interior in an immersive environment, a designer and an ergonomist can work together at the same time, the first one being in the position of the customer in the virtual vehicle, the second one taking a step back from the vehicle to measure and validate the ergonomics. In a co-located synchronous collaborative environment, the use of powerwall or CAVE-type systems is interesting compared to immersive headsets. Indeed, these systems allow collaborators to exchange while physically seeing each other, thus without the need to represent the others virtually, unlike immersive headsets, which shield from the real world (Fig. 8). However, the co-located simultaneous display of a digital mock-up in different representations is only possible if the projection systems have the capacity, which is still rarely the case, for reasons of cost and complexity of implementation. For this purpose, several solutions exist:

Other possibilities include asymmetric setups, i.e., different devices used in the same session, for example, a VR device and an AR device, or an immersive device and a control screen, which is particularly suitable to train operators for instance (see also Subsect. 4.3 and Fig. 9). In such a situation, the trainer not only sees the actions of the trainee in the virtual environment, but can also trigger events or scenarios thanks to a dedicated interface.

In the maintenance field, collaboration using immersive technologies has proven to be effective [8, 110]. The main point is collaboration between distant people, synchronously or asynchronously: an operator stands in front of a machine to be maintained wearing augmented or mixed reality glasses, or holding a tablet PC, and shares his/her environment with an expert located in a distant office who provides guidance through voice and augmented indications sent on the operator’s glasses or tablet. The key issue is to define the right amount of indications to display to the operator and the suitable interaction methods not to lead to cognitive overload [93].

One important aspect to consider when developing collaborative engineering in virtual environments relates to the management of data. Since each field generates its own model and representation using domain-specific tools, interoperability shall be provided to ensure data are correctly synchronized and integrated into the collaborative digital mock-up [4], and usable in immersive environments.

Immersive collaborative environments have great potential for learning and teaching use cases. They can be used not only for acquiring knowledge for a particular domain and enhancing motor-coordination and physical skills, but can also improve communication, collaboration, soft skills, and many more [34]. Further benefits of immersive teaching and learning (depending on the use case) may be the increment of motivation and concentration, saving of time and costs as well as higher safety and health protection, when the real environment is too dangerous [36]. Regarding the learning goals, technology setup, and implemented features, there are many ways to manifest those benefits. An early study showed that the immersive 3D learning environment helped participants feel part of a group and that the environment was an effective way to foster social interaction and group dynamics [73]. Communicating while in an immersive learning environment enables educational methodologies such as situated learning and peer/pair learning. Soft skills that are trained in CVEs are teamwork, problem-solving, decision-making, management, leadership competencies, and others. The networked virtual environment emulates the kind of collaboration, that is usual for the real world and can benefit collaborative and active learning [15]. An example for a multi-user VR environment is the vocational training for anaphylactic shock, which enables the critical paramedical cases, which happen too rarely, to be trained in a safe collaborative environment [94]. The industrial training, provided as an example by Sobota et al. [100], shows that using a CVE can deliver a practical training without the need to allocate real workspace or even instructors, which leads to saving many resources.

From a technical perspective, an educational setup can use powerwalls or CAVE systems, where a group of learners with or without a teacher can observe, interact and discuss the projected virtual environment. Learners can collaborate co-located in a team, in order to solve a specific task, discover mistakes, solve a problem or make decisions together. For example, the MIT Media Lab presents a collaborative co-located environment called Electrostatic Playground, where learners can explore and discover principles of electrostatics through experimentation [35]. Furthermore, the asymmetric collaboration between CAVE and VR or AR headsets in remote locations can also be applied efficiently to learning [24, 84].

Immersive environments can also provide a good communication platform for learning a foreign language, especially between students from different countries and cultures. In the context of communication, the use of VR can overcome language barriers [80]. If trainees do not understand the information verbally, an immersive environment provides a visual representation. Furthermore, the learning environment can be supported by automatic real-time translation algorithms to enhance the collaboration.

While collaborative virtual environments for educational purposes have numerous benefits, there are also some risks to be considered, such as safety (risks of mental health issues due to prolonged exposure to VR or heavy headsets used by young children) or cybersecurity (risks of not permitted access to personal information and biometric data from third parties). Nevertheless, researchers are working to address these limitations, and we will see more and more widespread use of CVE applications in the education area in the future.