Submitted Successfully!
To reward your contribution, here is a gift for you: A free trial for our video production service.
Thank you for your contribution! You can also upload a video entry or images related to this topic.
Version Summary Created by Modification Content Size Created at Operation
1 Summary of limitations and recommendations for using brain-computer interfaces in virtual reality. + 2285 word(s) 2285 2020-11-19 10:04:24 |
2 format change -21 word(s) 2264 2020-11-20 05:27:25 |

Video Upload Options

Do you have a full video?

Confirm

Are you sure to Delete?
Cite
If you have any further questions, please contact Encyclopedia Editorial Office.
Cattan, G.; Andreev, A.; Visinoni, E.; Cattan, G. Recommendations for Integrating P300-based BCI. Encyclopedia. Available online: https://encyclopedia.pub/entry/3118 (accessed on 15 June 2024).
Cattan G, Andreev A, Visinoni E, Cattan G. Recommendations for Integrating P300-based BCI. Encyclopedia. Available at: https://encyclopedia.pub/entry/3118. Accessed June 15, 2024.
Cattan, Grégoire, Anton Andreev, Etienne Visinoni, Grégoire Cattan. "Recommendations for Integrating P300-based BCI" Encyclopedia, https://encyclopedia.pub/entry/3118 (accessed June 15, 2024).
Cattan, G., Andreev, A., Visinoni, E., & Cattan, G. (2020, November 19). Recommendations for Integrating P300-based BCI. In Encyclopedia. https://encyclopedia.pub/entry/3118
Cattan, Grégoire, et al. "Recommendations for Integrating P300-based BCI." Encyclopedia. Web. 19 November, 2020.
Recommendations for Integrating P300-based BCI
Edit

The integration of a P300-based brain-computer interface (BCI) into virtual reality (VR) environments is promising for the video games industry. However, it faces several limitations, mainly due to hardware constraints and limitations engendered by the stimulation needed by the BCI. The main restriction is still the low transfer rate that can be achieved by current BCI technology, preventing movement while using VR. Adventure and simulation games, appear to be the best candidates for designing an effective VR game enriched by BCI technology.

BCI VR design game design brain–computer interface virtual reality virtual environment head-mounted display HMD

1. Introduction

A video game can be defined as “a mental contest, played with a computer according to certain rules, for amusement, recreation, or winning a stake”.[1] It has also been defined briefly as “story, art, and software”.[1] Sometimes, for example, in serious games, amusement is not the main goal. However, to date, amusement still plays a major role in the video game industry. Although by different means, virtual reality (VR) and brain-computer interface (BCI) are both excellent candidates for enhancing the possibilities of entertainment and satisfaction in video games. Indeed, both enhance immersion and it is a common belief that this encourages the amusement feeling. The concept of immersion was defined in Brown and Cairns,[2] observing that everybody may enjoy a game with immersion, even if the gaming control seems to play the main role in the user’s enjoyment. According to Zyda,[1] this immersion feeling is created by computer graphics, sound, haptics, affective computing, and advanced user interfaces that increase the sense of presence. Virtual reality refers to a collection of devices and technologies enabling the end-user to interact in 3D[3], e.g., spatialized sounds and haptic gloves (for example Dexmo, Dexta Robotics, Shenzhen, China). Steuer[4] emphasizes the particular type of experience that is created by VR. Such experience is named telepresence, defined as the experience of presence in an environment by means of a communication medium,[4] joining the concept of presence.[1] A BCI can also enhance the feeling of presence in the virtual world since it can replace or enhance mechanical inputs. According to Brown and Cairns,[2] immersive games are played using three kinds of inputs: visual, auditory, and mental; since a BCI may transform ‘mental’ signals into input commands, such an interface may play a unique role in the mentalization process involved in the feeling of immersion. However, considering the limitations of a BCI system (to be analyzed later), it is still not clear to what extent current BCI technology may improve immersion. As pointed out by Brown and Cairns,[2] “engagement, and therefore enjoyment through immersion, is impossible if there are usability and control problems”.

An element of amusement derives from the originality and futuristic aspect of BCI technology as compared to other traditional inputs, like a mouse, a joystick, or a keyboard. Nonetheless, as it often happens in the technological industry, BCI technology risks being dropped by the public if the improvement it brings is not worthwhile compared to the effort needed for its use. Virtual reality has already enjoyed the “wow-factor” and VR systems tend to be employed nowadays in commercial events especially for raising this effect (Feel Wimbledon by Jaguar, Coca Cola’s Santa’s Virtual Reality Sleigh Ride, McDonald’s Happy Meal VR Headset, and Ski App, Michelle Obama’s VR Video, XC90 Test Drive by Volvo, etc.) (http://mbryonic.com/best-vr/). The “wow-factor” is defined in the Cambridge dictionary as “a quality or feature of something that makes people feel great excitement or admiration”, and was previously studied in the domain of marketing or education[5][6]. The recent development of dedicated VR headsets, that is, head-mounted-devices (HMDs, e.g., the Oculus, Facebook, Menlo Park, CA, USA; HTC Vive, HTC, Taoyuan, Taiwan; Google Cardboard, Google, Mountain View, CA, USA) has paved the way to the commercialization of combined BCI+VR technology. Indeed, HMDs provide an already build-in structure that can support the embedding of EEG (electroencephalography) electrodes, which are needed for the BCI. The Neurable Company (Cambridge, MA, USA) has recently announced a product combining an HTC Vive (Taoyuan, Taiwan) with an EEG cap. The company claims to develop an everyday BCI and successfully raised 6 million dollars for this purpose (https://www.forbes.com/sites/solrogers/2019/12/17/exclusive-neurable-raises-series-a-to-build-an-everyday-brain-computer-interface/). The HTC Vive (Taoyuan, Taiwan), as well as other HMDs such as the SamsungGear (Samsung, Seoul, Korea) or the Oculus Quest (Facebook, Menlo Park, CA, USA), use inboard electronics, thus herein we refer to them as active devices. On the contrary, passive HMDs consist of a simple mask with lenses in which a smartphone is inserted (Figure 1). Passive HMDs are particularly promising for the BCI+VR field since they are very affordable and smartphones are nowadays ubiquitous. In addition, passive HMDs, like SamsungGear and Oculus Quest, are easily transported since they do not require the use of additional materials, such as personal computers or external sensors.

Figure 1. The SamsungGear HMD (a) can be used in passive mode (inserting a smartphone without plugging it into the mask through the micro-USB port) or active mode (with on-board electronic, mainly the gyroscope, supplied through the micro-USB port). The Neurable headset (b) combines EEG with the HTC Vive (c), an active VR headset linked to a powerful computer. The Oculus Quest (d) is an active device that offers similar functionalities to the HTC Vive without the need for external sensors, thus it is easily transported.

Prototypes of BCI-based video games already exist.[7][8][9][10][11][12][13][14][15][16][17][18] They are mainly based on three BCI paradigms: the steady-state-evoked-potential (SSVEP), P300 event-related potentials (ERP), and mental imagery (MI). The first two necessitate sensorial stimulation of the user, usually visual, and are defined as synchronous because the application decides when to activate the stimulation so that the user can give a command.[19] In this article, we focus on P300-based BCIs. As compared to MI-based BCIs, P300-based BCIs require a shorter training, achieve a higher information transfer rate (amount of information sent per unit of time), and allow a higher number of possible commands.[20][21] As compared to SSVEP-based BCIs they feature a lower information transfer rate. However, the flickering used for eliciting SSVEPs is annoying and tiring, besides presenting an increased risk of eliciting epileptic seizures.[22] Modern P300-based BCI was introduced in 1988 by Farwell and Donchin as a means of communication for patients suffering from the ‘locked-in’ syndrome.[23] They were probably influenced by the prior work of Vidal, who proposed in the ’70s a conceptual framework for online ERP-based BCI,[24] coining, by the way, the expression brain-computer interface.[25][26] P300-based BCIs are based on the so-called oddball paradigm. The oddball paradigm is an experimental design consisting of the presentation of discrete stimuli successively; most are neutral (non-TARGET) and a few (rare) are TARGET stimuli.[27] In the case of P300-based BCI, items are flashed on the screen, typically in groups. A sequence of flashes covering all available items is named a repetition. The goal of the BCI is to analyze the ERPs in one or more repetitions to individuate which item has produced a P300, a positive ERP that appears around 300–600 ms after the item the user wants to select (TARGET) has flashed. The typical accuracy of P300-based BCIs has risen over the past years from about around 75% after 15 repetitions of flashes[28] to around 90% after three repetitions using modern machine learning algorithms based on Riemannian geometry.[29][30][31][32]  In practice, this means that at least one second is necessary for such BCI to issue a command, but more may be needed to issue reliable commands.

We anticipate that the integration of BCI in VR games thanks to the development of integrated HMD-EEG[33][34][35] devices will foster the acceptance of this technology by both the video game industry and gamers, thus pushing the technology into the real world. The development of a concrete application for the large public faces several limitations, though. In the domain of virtual reality, motion sickness appears to be one of the most severe limitations. However, this limitation seems relatively weak in comparison to those risen by the BCI system. Above all, BCIs are often unsightly, and the electrodes are not easy to use. Also, users in virtual reality may move a lot and this jeopardizes the quality of the EEG signal.

2. Type of Game Recommendations

Marshall and al.[36] studied the possible applications of BCI technology depending on the type of game. In the following, we review the recommendations given by these authors.

  • Real-Time Strategy (RTS) games are too complex and need continuous control, thus P300 does not suit them. In RTS games, P300 can still be used if restricting it to non-critical control aspects. Generally, however, strategy games are not particularly adapted to the VR context, as only 7% of VR games are of this kind (Figure 2a). For these games, the recommendation is to have a third-person point of view. For example, the player’s avatar controls a map representing the game field and this map is an object in a virtual room. Although it is not a strategy game, MOSS (Polyarc, Seattle, WA, USA) illustrates the use of a third-person point of view in VR.
  • Roleplay games (RPG) are also problematic for P300-based BCI because of their complexity. The general recommendations are the same as for RTS. The RPG should be turn-based and the BCI should be restricted to minor aspects of the game. An example is “Alpha wow”,[9] where the user’s mental state is used to change the avatar’s behavior in the virtual world.
  • Action games are the most popular type of game employing BCI technology. This is surprising since action games often include fast-moving gameplay. For this reason, the use of BCI is not recommended in action games without adaptation.
  • Sports games meet the same requirements as action games.[36] They often require fast-moving gameplay and continuous control. Therefore, we do not recommend the use of BCI for sport games without adaptation, such as the one given as an example in Table 1. In VR, sports games represent a moderate percentage of the games (9%, Figure 2a).
  • Puzzle games are very well suited for P300-based applications. They should be turn-based, allowing users to make simple choices at their own pace. The use of existing popular puzzles helps players because they are already familiar with the game’s rules. However, the problem is the same as for strategy games and board games in general, i.e., it is not very useful to adopt a 3D perspective with a board game (puzzle games represent only 2% of VR games—Figure 2a). We suggest the use of the same workaround as for strategy games and to use a third-person point of view. In such a scenario, puzzle games may be presented as a board game inside a virtual room. Another idea is to design a puzzle in 3D allowing the player to move the pieces in all directions (and, why not, to move inside the puzzle itself).
  • Adventure games are well suited for P300-based BCIs, if the player is given a set of limited options within a given time. 
  • Simulation games (for training or education purposes for example) are also well suited for P300-based BCI, especially in the case of management simulation. Simulation games should feature slow gameplay, allowing the player to adjust and learn how to control the BCI. In addition, simulation games are not based on “score”. Therefore the player can relax and obtain better performance using BCI control 

Furthermore, we should consider the means of interactions with the VE, that is, tracking of the user’s movements in a room or use of traditional inputs (joystick, mouse, keyboard) while sitting. Indeed, room-scaled VR games seem less suitable for BCI, as rotating the head or walking through the game area could perturb the EEG signal due to muscular artifacts or swinging of the electrode cables (as exposed in Section 3.2). However, it could be challenging to design a VR game using traditional inputs. Indeed, according to the Steam Platform (Valve Corporation, Bellevue, WA, USA), a ubiquitous offering platform for games, only 13% of VR games are played on a chair with a keyboard or a mouse.

In conclusion, the P300 paradigm suits well turn-based strategy games (board games such as chess or some PC game as Civilization or Heroes of Myths’ and Magic). Adventure and simulation games appear to be the most adapted types of games for a BCI-VR game (Figure 2b), provided that they implement slow-game play and restrict the player’s movements. Non-surprisingly, the over-representation of adventure and simulation in VR games has not changed since our first investigation in 2017,[37] outlining a stable interest of the general public for this type of game. Interestingly, independent studios are very well represented in the VR versus PC and console industry. In our opinion, this can be explained by (1) the early integration of VR into game editors such as Unity 3D (Unity, San Francisco, CA, USA) or Unreal Engine (Epic Games, Cary, NC, USA) and (2) a more willing to experiment new technologies (https://www.vrfocus.com/2020/05/why-now-is-the-time-for-aaa-studios-to-consider-vr/; https://labusinessjournal.com/news/2015/jun/17/independent-virtual-reality-studios-benefit-early-/).

Figure 2. Benchmark of the type of game in VR and BCI. (a) The repartition of VR games by type according to the Steam Platform (Valve Corporation, Bellevue, US) – see https://store.steampowered.com/search/?tags=597&category1=998&vrsupport=402. (b) Classification of the different types of games in regard to the previously exposed recommendations, and their representation in the VR market. The color code indicates either the type of game is suitable for VR or BCI. The suitability for VR or BCI increases from right to left.

References

  1. Zyda, M. From visual simulation to virtual reality to games. Computer 2005, 38, 25–32.
  2. Brown, E.; Cairns, P. A grounded investigation of game immersion. In Proceedings of the Extended Abstracts of the 2004 Conference on Human Factors and Computing Systems-CHI’04; ACM: New York, NY, USA, 2004; pp. 1297–1300.
  3. Harvey, D. Invisible Site: A Virtual Sho. (George Coates Performance Works, San Francisco, California). Variety 1992, 346, 87.
  4. Steuer, J. Defining Virtual Reality: Dimensions Determining Telepresence. J. Commun. 1992, 42, 73–93.
  5. Tokman, M.; Davis, L.M.; Lemon, K.N. The WOW factor: Creating value through win-back offers to reacquire lost customers. J. Retail. 2007, 83, 47–64.
  6. Bamford, A. The Wow Factor: Global Research Compendium on the Impact of the Arts in Education; Waxmann Verlag: Münster, Germany, 2006; ISBN 978-3-8309-6617-3.
  7. Lécuyer, A.; Lotte, F.; Reilly, R.B.; Leeb, R.; Hirose, M.; Slater, M. Brain-Computer Interfaces, Virtual Reality, and Videogames. Computer 2008, 41, 66–72.
  8. Andreev, A.; Barachant, A.; Lotte, F.; Congedo, M. Recreational Applications of OpenViBE: Brain Invaders and Use-the-Force; John Wiley & Sons: Hoboken, NJ, USA, 2016; Volume 14, ISBN 978-1-84821-963-2.
  9. Van De Laar, B.; Gurkok, H.; Bos, D.P.-O.; Poel, M.; Nijholt, A. Experiencing BCI Control in a Popular Computer Game. IEEE Trans. Comput. Intell. AI Games 2013, 5, 176–184.
  10. Mühl, C.; Gürkök, H.; Bos, D.P.-O.; Thurlings, M.E.; Scherffig, L.; Duvinage, M.; Elbakyan, A.A.; Kang, S.; Poel, M.; Heylen, D. Bacteria Hunt. J. Multimodal User Interfaces 2010, 4, 11–25.
  11. Angeloni, C.; Salter, D.; Corbit, V.; Lorence, T.; Yu, Y.C.; Gabel, L.A. P300-based brain-computer interface memory game to improve motivation and performance. In Proceedings of the 2012 38th Annual Northeast Bioengineering Conference (NEBEC), Philadelphia, PA, USA, 16–18 March 2012; pp. 35–36.
  12. Kaplan, A.Y.; Shishkin, S.L.; Ganin, I.P.; Basyul, I.A.; Zhigalov, A.Y. Adapting the P300-Based Brain–Computer Interface for Gaming: A Review. IEEE Trans. Comput. Intell. AI Games 2013, 5, 141–149.
  13. Pires, G.; Torres, M.; Casaleiro, N.; Nunes, U.; Castelo-Branco, M. Playing Tetris with non-invasive BCI. In Proceedings of the 2011 IEEE 1st International Conference on Serious Games and Applications for Health (SeGAH), Braga, Portugal, 16–18 November 2011; pp. 1–6.
  14. Liao, L.-D.; Chen, C.-Y.; Wang, I.-J.; Chen, S.-F.; Li, S.-Y.; Chen, B.-W.; Chang, J.-Y.; Lin, C.-T. Gaming control using a wearable and wireless EEG-based brain-computer interface device with novel dry foam-based sensors. J. Neuroeng. Rehabil. 2012, 9, 5.
  15. Edlinger, G.; Guger, C. Social Environments, Mixed Communication and Goal-Oriented Control ApplicationUsing a Brain-Computer Interface. In Universal Access in Human-Computer Interaction. Users Diversity;Springer: Berlin/Heidelberg, Germany, 2011; pp. 545–554.
  16. Gürkök, H. Mind the Sheep! User Experience Evaluation & Brain-Computer Interface Games; University of Twente:Enschede, The Netherlands, 2012.
  17. Congedo, M.; Goyat, M.; Tarrin, N.; Ionescu, G.; Varnet, L.; Rivet, B.; Phlypo, R.; Jrad, N.; Acquadro, M.;Jutten, C. “Brain Invaders”: A prototype of an open-source P300- based video game working with theOpenViBE platform. In Proceedings of the 5th International Brain-Computer Interface Conference 2011(BCI 2011), Styria, Austria, 22–24 September 2011; pp. 280–283.
  18. Ganin, I.P.; Shishkin, S.L.; Kaplan, A.Y. A P300-based Brain-Computer Interface with Stimuli on MovingObjects: Four-Session Single-Trial and Triple-Trial Tests with a Game-Like Task Design. PLoS ONE 2013,8, e77755.
  19. Wolpaw, J.; Wolpaw, E.W. Brain-Computer Interfaces: Principles and Practice; Oxford University Press: New York, NY, USA, 2012; ISBN 978-0-19-538885-5.
  20. Zhang, Y.; Xu, P.; Liu, T.; Hu, J.; Zhang, R.; Yao, D. Multiple Frequencies Sequential Coding for SSVEP-BasedBrain-Computer Interface. PLoS ONE 2012, 7, e29519.
  21. Sepulveda, F. Brain-actuated Control of Robot Navigation. In Advances in Robot Navigation; IntechOpen:London, UK, 2011; Volume 8, ISBN 978-953-307-346-0.
  22. Fisher, R.S.; Harding, G.; Erba, G.; Barkley, G.L.; Wilkins, A. Epilepsy Foundation of America Working GroupPhotic- and pattern-induced seizures: A review for the Epilepsy Foundation of America Working Group.Epilepsia 2005, 46, 1426–1441.
  23. Farwell, L.A.; Donchin, E. Talking off the top of your head: Toward a mental prosthesis utilizing event-relatedbrain potentials. Electroencephalogr. Clin. Neurophysiol. 1988, 70, 510–523.
  24. Vidal, J.J. Real-time detection of brain events in EEG. Proc. IEEE 1977, 65, 633–641.
  25. Lotte, F.; Nam, C.S.; Nijholt, A. Introduction: Evolution of Brain-Computer Interfaces; Taylor & Francis(CRC Press): Abingdon-on-Thames, UK, 2018; pp. 1–11. ISBN 978-1-4987-7343-0.
  26. Vidal, J.J. Toward Direct Brain-Computer Communication. Annu. Rev. Biophys. Bioeng. 1973, 2, 157–180.
  27. Squires, N.K.; Squires, K.C.; Hillyard, S.A. Two varieties of long-latency positive waves evoked byunpredictable auditory stimuli in man. Electroencephalogr. Clin. Neurophysiol. 1975, 38, 387–401.
  28. Guger, C.; Daban, S.; Sellers, E.; Holzner, C.; Krausz, G.; Carabalona, R.; Gramatica, F.; Edlinger, G.How many people are able to control a P300-based brain-computer interface (BCI)? Neurosci. Lett. 2009,462, 94–98.
  29. Congedo, M. EEG Source Analysis, Habilitation à Diriger des Recherches; Université de Grenoble: Grenoble, France, 2013.
  30. Barachant, A.; Bonnet, S.; Congedo, M.; Jutten, C. Multiclass brain-computer interface classification by Riemannian geometry. IEEE Trans. Biomed. Eng. 2012, 59, 920–928.
  31. Barachant, A.; Congedo, M. A Plug&Play P300 BCI Using Information Geometry. ArXiv14090107 Cs Stat.August 2014. Available online: http://arxiv.org/abs/1409.0107.
  32. Congedo, M.; Barachant, A.; Bhatia, R. Riemannian geometry for EEG-based brain-computer interfaces; a primer and a review. Brain-Comput. Interfaces 2017, 4, 155–174.
  33. Cattan, G.H.; Andreev, A.; Mendoza, C.; Congedo, M. A comparison of mobile VR display running on an ordinary smartphone with standard PC display for P300-BCI stimulus presentation. IEEE Transactions on Games 2019, 1–1, doi:10.1109/TG.2019.2957963.
  34. Käthner, I.; Kübler, A.; Halder, S. Rapid P300 brain-computer interface communication with a head-mounted display. Frontiers in Neuroscience 2015, 9, 207, doi:10.3389/fnins.2015.00207.
  35. Cattan, G.; Andreev, A.; Maureille, B.; Congedo, M. Analysis of tagging latency when comparing event-related potentials; Gipsa-Lab ; IHMTEK: Grenoble, 2018;
  36. Marshall, D.; Coyle, D.; Wilson, S.; Callaghan, M. Games, Gameplay, and BCI: The State of the Art. IEEE Trans. Comput. Intell. AI Games 2013, 5, 82–99.
  37. Cattan, G. De la Réalisation d’une Interface Cerveau-Ordinateur pour Une Réalité Virtuelle Accessible au Grand Public. Ph.D. Thesis, Université Grenoble Alpes, Grenoble, France, 2019.
More
Information
Subjects: Others
Contributors MDPI registered users' name will be linked to their SciProfiles pages. To register with us, please refer to https://encyclopedia.pub/register : , , ,
View Times: 657
Revisions: 2 times (View History)
Update Date: 10 Dec 2020
1000/1000
Video Production Service