Improving dermatologic teaching is a challenge, as visual memory is difficult to maintain without constant exposure. As a result, it has been reported that the diagnostic accuracy of common skin diseases is poor by primary care physicians [12]. The use of 3D silicon-made phantoms greatly improves diagnostics skills and knowledge retention in comparison to standard 2D images [13]. Therefore, it is expected that AR should exhibit similar effects [8]. Nonetheless, recent studies have demonstrated ambiguous results with minor differences in knowledge gain and slightly better knowledge retention for the group using AR-based tools in comparison to textbooks [7]. This outcome could potentially be improved by coupling AR with haptic feedback, as is the case in most surgical training tools.

A potential application that could prove to be beneficial in training the next generation of medical experts would include realistic-looking 3D models displaying skin diseases and irregularities (see Figure 1). These models would contain high resolution clinical textures that are anonymized on newly generated 3D models with anatomical proportions, all of which could be run on smartphones/tablets or VR goggles. The environmental conditions (like lighting conditions) could be adapted for a more realistic experience. In addition, such models could be expanded by using synthetic clinical images, to reduce data scarcity and poor training regarding rare dermatological diseases and certain Fitzpatrick skin types. For example, incidence of melanoma is relatively rare among black populations, where survival rates lag behind the rates for non-Hispanic white populations in the US [14]. Such issues have recently been highlighted in the press [15]. 

Aside from health professionals, some studies have investigated AR/VR as an educational tool for patients or the public to increase their awareness of skin disease prevention. The group Elke Hacker in Australia has developed a virtual reality (VR) game containing preventative skin cancer messages for youth and young adults [17]. This was based on the fact that despite successful public campaigns in the 1980’s, high rates of sunburn continue to be reported among the young population.

In another application of VR in dermatology, we created the VR app Virtual Derm, which places students in a virtual practice for dermatology [18]. The app runs on standard Android smartphones, making it accessible to most students. A total of 100 patient vignettes can be solved by students, making it a unique learning opportunity (see Figure 2).

Dermatologic surgery is the practice of dermatology that specializes in surgical procedures to repair or improve the function and appearance of the skin [19]. While plastic surgery greatly benefited from AR/VR technology [20], few applications have been reported in dermatologic surgery. Aside from surgical procedure training [21], AR and VR have been implemented in Mohs surgery to reduce patient anxiety [22] or as a surgical assistance tool [23].

For patients, VR has been used to reduce pre- and intraoperative anxiety. The tools employed in this study were unspecific VR sequences which could be used to reduce anxiety under any circumstance. It has been shown that almost all anxiety-related parameters could be improved in the context of surgery [22]. More specifically, VR could be used as tool for patient education, as many patients do not have a clear understanding what a surgical procedure entails, such as using a flap for the reconstruction of a surgical defect. However, this has not been investigated and may also induce anxiety in some patients where too much detail about a surgical procedure is given.

For physicians, VR has been used as a training tool in many surgical specialties and has great potential in dermatologic surgery. The more realistic appearance of a VR image, combined with haptic sensations, can be used to train dermatologic surgical interventions in the same way as in other surgical fields, notably plastic surgery. AR has been successfully used by one group to improve the communication between surgeons and pathologists in cases of Mohs surgery, where the pathology slides were not necessarily read by the surgeon themselves [23]. In addition, AR has great potential for other aspects of Mohs surgery and dermatologic surgery. After acquiring the theoretical background and training with artificial models, a surgical procedure is best learnt by performing the procedure on a real patient. In most training programs, this is performed under the direct supervision of a senior faculty member who is present at the operating table. However, due to shortcomings in staff resources and planning, this is not always possible and junior surgeons sometimes find themselves performing an operation that is above their level of confidence alone. AR could allow a senior faculty member to follow an operation remotely and give very precise and practical instructions to the junior surgeon at the table, with AR images superimposed on the surgical site to guide the surgery. In larger institutions, this would also allow senior staff resources to be used more efficiently, as one experienced surgeon could follow several surgeries at the same time.

Artificial intelligence (AI) has been productively utilized for several decades in dermatology, particularly for skin cancer detection. It has been reported that convolutional neural networks (CNNs) achieve dermatologist-level classification of cutaneous lesions using both dermoscopic and non-dermoscopic images [24]. AI has proven very useful when combined with VR reconstruction of the body surface. In a prominent example, the Canfield^® Vectra WB360 device combines 2D photographs of 46 stereo-vision pods, resulting in a full-body VR reconstruction that can be turned and manipulated at will. The machine further analyzes all photographs with ML-based segmentation and classification, looking for naevi and other cutaneous lesions. All naevi can be analyzed by artificial intelligence for their probability of being malignant. The information on these lesions is saved, with the position of each lesion on the body map, and upon repeated patient use of the Vectra device, changes are tracked over time. By utilizing this technology, newly developing melanomas are actively being found.

AI-based smartphone applications, or “apps”, are being successfully developed as practical tools to improve diagnostics [25,26,27,28], despite some limitations in terms of accuracy for melanoma screening [27].

The use of AR/VR in dermatology could strongly benefit from haptic feedback. Haptic feedback, or “haptics”, refers to manual interaction with the environment, and could be used to relay force and tactile information to the user [29]. Indeed, sensory information such as skin temperature, roughness or stiffness would provide valuable additional information to support diagnostic evaluation [30]. In this direction, Kim and co-workers developed a method to convert a single image to a 3D haptic surface and render the generated haptic surface in real-time [31,32].