21-02-2011, 02:46 PM
Brain-Computer Interfaces, Virtual Reality, and Videogames
Far beyond science-fiction clichés and the image of a person connected to cyberspace via direct cerebral
implants as in The Matrix, brain-computer interfaces (BCIs) can offer a new means of playing videogames
or interacting with 3D virtual environments (VEs).
Only in recent years have research groups been attempting to connect BCIs and virtual worlds.
However, several impressive prototypes already exist that enable users to navigate in virtual scenes or
manipulate virtual objects solely by means of their cerebral activity, recorded on the scalp via
electroencephalography (EEG) electrodes. Meanwhile, virtual reality (VR) technologies provide
motivating, safe, and controlled conditions that enable improvement of BCI learning as well as the
investigation of the brain responses and neural processes involved.
STATE OF THE ART
VR technologies and videogames can be powerful BCI companions. Researchers have shown that BCIs
provide suitable interaction devices for VR applications [10] and videogames [6]. On the other hand, the
community now widely accepts that VR is a promising and efficient medium for studying and improving
BCI systems.
Brain-computer interaction with virtual worlds
Interactions with VE can be decomposed into elementary tasks [1] such as navigating to change the
viewpoint or selection and manipulation of virtual objects.
In virtual worlds, current BCI systems can let users change the camera position in a VE toward the left
or right by using two different brain signals, such as left- or right-hand motor imagery (MI) or two steadystate
visual-evoked potentials (SSVEP) at different frequencies. MI-based BCIs have also been used to
control the steering of a virtual car [2], explore a virtual bar [10], or move along a virtual street [3] or
through a virtual flat [7]. These BCIs typically provide the user with one to three commands, each
associated with a given task.
Concerning selection and manipulation of virtual objects, developers base most BCIs on P300 or SSVEP
signals. In these applications, virtual objects generally provide a stimulus that triggers a specific and
recognizable brain signal that draws the user’s attention to the associated object to select and manipulate it.
Those BCIs let the user turn on and off devices such as a virtual TV or lamp using the P300 [4], or
manipulate more complex objects such as a virtual avatars using SSVEP
Virtual reality for studying and improving BCI
Researchers can use VR to study and improve brain-computer interaction. The technology also helps
researchers perform safe and perfectly controlled experiments. For example, it has enabled the simulation
of wheelchair control with a BCI [3] and various BCI groups have used it to study how users would react
while navigating in a complex 3D environment using a BCI in close to real-life conditions [7][9].
Several studies have compared feedback consisting of classical 2D displays with feedback consisting of
entertaining VR applications [2][7]. These studies show that users’ performance ranked higher with VR
feedback than with simple 2D feedback. Moreover, evidence suggests that the more immersive the VR
display, the better users perform [7][10]. Even though some observations await confirmation, VR appears
to shorten BCI learning and increase users’ performance by increasing their motivation.
TYPICAL APPLICATIONS
Several universities and laboratories have pursued the creation of more compelling interaction with
virtual worlds using BCI, including University College Dublin, MediaLabEurope, Graz University of
Technology, University College London, University of Tokyo and INRIA.
MindBalance videogame
Researchers at University College Dublin and MediaLabEurope have created MindBalance [5], a
videogame that uses BCI to interact with virtual worlds. As Figure 1 shows, the game involves moving an
animated 3D character within a virtual environment. The objective is to gain one-dimensional control of the
character’s balance on a tightrope using only the player’s EEG. The developed BCI uses the SSVEP
generated in response to phase-reversing checkerboard patterns. The SSVEP simplifies the signalprocessing
methods dramatically so that users require little or no training. The game positions a checkerboard on either side of the character. These checkerboards are phasereversed
at 17 and 20 Hz. Each game begins with a brief calibration period. This requires the subject to
attend to the left and right checkerboards, as indicated by arrows, for 15 seconds each. The system uses the
recorded data to calibrate the BCI and adapt its parameters to the current player’s EEG. This process
repeats three times.
When playing the game, the user must control the animated character, which is walking a tightrope
while being subjected to random movements to the left and right. If the user does not accurately attend to
the correct side to control the character after initially losing balance (first degree), the character will move
to a more precarious (second degree) state of instability, then, progressively, to an unrecoverable state
(third degree), at which point the user falls.
For correct user control, the animated character will move up a degree of balance until perfectly upright,
allowing forward progress to resume. Audiovisual feedback streams into the user’s file, providing
information on the character’s stability. The visual feedback shows the degree of inclination in relation to
the tightrope.
The BCI’s performance proved to be robust in resisting distracting visual stimulation in the game’s
visually rich environment and relatively consistent across six subjects, with 41 of 48 games successfully
completed.
The average real-time control accuracy across subjects was 89 percent. Some subjects achieved better
performance in terms of success in completing the game. This suggests that either practice or a more
motivated approach to stimulus fixation results in a more pronounced visual response.
Dual university collaboration
In a first experiment designed by researchers at the Graz University of Technology and the University
College London’s virtual reality laboratory, a tetraplegic subject mastered control of his wheelchair’s
simulated movements along a virtual street populated with 15 virtual characters
download full report
http://irisa.fr/bunraku/GENS/alecuyer/Le..._draft.pdf
Far beyond science-fiction clichés and the image of a person connected to cyberspace via direct cerebral
implants as in The Matrix, brain-computer interfaces (BCIs) can offer a new means of playing videogames
or interacting with 3D virtual environments (VEs).
Only in recent years have research groups been attempting to connect BCIs and virtual worlds.
However, several impressive prototypes already exist that enable users to navigate in virtual scenes or
manipulate virtual objects solely by means of their cerebral activity, recorded on the scalp via
electroencephalography (EEG) electrodes. Meanwhile, virtual reality (VR) technologies provide
motivating, safe, and controlled conditions that enable improvement of BCI learning as well as the
investigation of the brain responses and neural processes involved.
STATE OF THE ART
VR technologies and videogames can be powerful BCI companions. Researchers have shown that BCIs
provide suitable interaction devices for VR applications [10] and videogames [6]. On the other hand, the
community now widely accepts that VR is a promising and efficient medium for studying and improving
BCI systems.
Brain-computer interaction with virtual worlds
Interactions with VE can be decomposed into elementary tasks [1] such as navigating to change the
viewpoint or selection and manipulation of virtual objects.
In virtual worlds, current BCI systems can let users change the camera position in a VE toward the left
or right by using two different brain signals, such as left- or right-hand motor imagery (MI) or two steadystate
visual-evoked potentials (SSVEP) at different frequencies. MI-based BCIs have also been used to
control the steering of a virtual car [2], explore a virtual bar [10], or move along a virtual street [3] or
through a virtual flat [7]. These BCIs typically provide the user with one to three commands, each
associated with a given task.
Concerning selection and manipulation of virtual objects, developers base most BCIs on P300 or SSVEP
signals. In these applications, virtual objects generally provide a stimulus that triggers a specific and
recognizable brain signal that draws the user’s attention to the associated object to select and manipulate it.
Those BCIs let the user turn on and off devices such as a virtual TV or lamp using the P300 [4], or
manipulate more complex objects such as a virtual avatars using SSVEP
Virtual reality for studying and improving BCI
Researchers can use VR to study and improve brain-computer interaction. The technology also helps
researchers perform safe and perfectly controlled experiments. For example, it has enabled the simulation
of wheelchair control with a BCI [3] and various BCI groups have used it to study how users would react
while navigating in a complex 3D environment using a BCI in close to real-life conditions [7][9].
Several studies have compared feedback consisting of classical 2D displays with feedback consisting of
entertaining VR applications [2][7]. These studies show that users’ performance ranked higher with VR
feedback than with simple 2D feedback. Moreover, evidence suggests that the more immersive the VR
display, the better users perform [7][10]. Even though some observations await confirmation, VR appears
to shorten BCI learning and increase users’ performance by increasing their motivation.
TYPICAL APPLICATIONS
Several universities and laboratories have pursued the creation of more compelling interaction with
virtual worlds using BCI, including University College Dublin, MediaLabEurope, Graz University of
Technology, University College London, University of Tokyo and INRIA.
MindBalance videogame
Researchers at University College Dublin and MediaLabEurope have created MindBalance [5], a
videogame that uses BCI to interact with virtual worlds. As Figure 1 shows, the game involves moving an
animated 3D character within a virtual environment. The objective is to gain one-dimensional control of the
character’s balance on a tightrope using only the player’s EEG. The developed BCI uses the SSVEP
generated in response to phase-reversing checkerboard patterns. The SSVEP simplifies the signalprocessing
methods dramatically so that users require little or no training. The game positions a checkerboard on either side of the character. These checkerboards are phasereversed
at 17 and 20 Hz. Each game begins with a brief calibration period. This requires the subject to
attend to the left and right checkerboards, as indicated by arrows, for 15 seconds each. The system uses the
recorded data to calibrate the BCI and adapt its parameters to the current player’s EEG. This process
repeats three times.
When playing the game, the user must control the animated character, which is walking a tightrope
while being subjected to random movements to the left and right. If the user does not accurately attend to
the correct side to control the character after initially losing balance (first degree), the character will move
to a more precarious (second degree) state of instability, then, progressively, to an unrecoverable state
(third degree), at which point the user falls.
For correct user control, the animated character will move up a degree of balance until perfectly upright,
allowing forward progress to resume. Audiovisual feedback streams into the user’s file, providing
information on the character’s stability. The visual feedback shows the degree of inclination in relation to
the tightrope.
The BCI’s performance proved to be robust in resisting distracting visual stimulation in the game’s
visually rich environment and relatively consistent across six subjects, with 41 of 48 games successfully
completed.
The average real-time control accuracy across subjects was 89 percent. Some subjects achieved better
performance in terms of success in completing the game. This suggests that either practice or a more
motivated approach to stimulus fixation results in a more pronounced visual response.
Dual university collaboration
In a first experiment designed by researchers at the Graz University of Technology and the University
College London’s virtual reality laboratory, a tetraplegic subject mastered control of his wheelchair’s
simulated movements along a virtual street populated with 15 virtual characters
download full report
http://irisa.fr/bunraku/GENS/alecuyer/Le..._draft.pdf
