19/04/2017
1
PA199
Advanced Game Design
Lecture 8
Brain Computer Interfaces and Games
Bojan Kerous
Fotis Liarokapis
13th April 2017
Introduction
Brain–Computer Interface (BCI) or Brain–
Machine Interface (BMI), is a direct way of
communication between the brain and a
computer system.
Types of BCI’s
Invasive
Non-invasive
Types of BCI systems
fMRI
fNIRS
MEG EEG
BCI pipeline Principles of EEG
19/04/2017
2
Principles of EEG Principles of EEG
section of the skull
Representation of cortex
Electrode placement EEG signal
EEG-based BCI paradigm
Event-related (P300)
Sensorimotor rhythms
Steady State Visually Evoked Potentials (SSVEP)
Event-related (P300) P300 spellers
19/04/2017
3
Sensorimotor rhythms
C3 (Left) C4 (Right)
Left Hand
Movement
Right Hand
Movement
SSVEP
Commercial BCI Headsets
• Non-invasive BCI’s most commonly use EEG:
– Portability, low set-up cost, easy of use
• Low-cost BCI headsets are used the last 5-7
years
Initial Work
• Two non-invasive BCI headsets have been used
to control
– Lego Mindstroms NXT Robot
– A simple computer game
• Two prototypes
– First Prototype was based on the Neurosky Mindset
– Second Prototype was based on the Emotiv Epoc
headset
Neurosky Headset
• NeuroSky MindWave is a simplified version of
the traditional EEG technology
• The MindSet monitors:
– Electrical potential between the sensing electrode
• Positioned on the forehead
– Reference electrodes
• Positioned on the left earlobe
Neurosky Advantages
• Very easy to use
• No calibration is required
– Plug and play!
• Good support is provided
– SDK
19/04/2017
4
First prototype - Neurosky
• Attention and Meditation levels are used
• Values from the headset are passed to the
robot
– Through the dedicated computer
• The robot is instructed to accelerate based on
the attention levels of the user
Neurosky Drawbacks
• Since there is only one sensor in place,
separating brainwaves becomes a challenge
• Because the headset is not fastened to the
head, pronounced muscle movements, such as
yawning, facial expressions may result in a
momentary decrease in signal quality
Initial User Evaluation Initial User Evaluation .
Recorded Feedback
Positive Negative
 Interesting concept
 Could be very useful
 Lots of potential
 Can help increase peoples
concentration level
 Can help improve disabled
peoples lives
 Can help for brain
rehabilitation by triggering
the motor cortex of the
brain
 The feel of controlling it
with my mind was
awesome.
 Not being able to move
left/right
 Need some indication on
the PC as to the level of
thought
 Needed less distractions
 Difficult to keep the robot
stationary
 Too many outside stimuli
 Hard to remain calm
Second Prototype - Emotiv
• The system relies on a combination of Cognitive
and Facial/Muscular functions
• The Emotiv Development Kit was used to create
and train a new user profile
19/04/2017
5
Emotiv Epoc Headset
• Emotiv Epoc Headset has
14 wet sensors (and 2
reference sensors)
detecting brain signals and
facial expressions
Emotiv Epoc Headset .
• Sensors are placed on the
international 10-20 system,
which describes the
electrode placement on the
scalp for EEG tests or
experiments
User Training
• Emotiv needs a unique
user profile to be trained
to map users’ brain-activity
Control Panel
User Training .
• In a training session no
more than 1 min, user’s
skills are increasing
approximately from 45%-
70% for the ”push”
action EmoKey
Game Control
• Each trained thought from the BCI headset is
treated as a keystoke
• That communicates with Emotiv EmoEngine as a
third-party application by using the Emotiv API
• The Emotiv EmoEngine refers to the logical
abstraction of the functionality that Emotiv
provides in edk.dll
BCIs and Computer Games
19/04/2017
6
Methodology
• Profile training using Control Panel for 60s
(push/pull actions plus blink calibration)
– Navigating the 3D robot inside the maze to a predefined
waypoint (increasing users cognitive
workload)
• A second training session of 60s
– Interacting with RomaNova
• Evaluation form completion and feedback
interview
3D Maze Game
• The 3D environment
has been designed
using the ‘Unity 3D’
game engine with a
3D reconstruction of
the robot and a
simple 3D maze
Game Elements
• 3D Robot
– The LEGO NXT Mindstorms Robot 3D
model has been used with a simple
wheel animation, interacting with
basic physics
• Maze walls
– The maze is made by a set of heterosized
blocks
Roma Nova Project
• Seeking to advance information transfer
through immersive ‘living background’
• Partners:
– Serious Games Institute
– University of Toulouse
Brain Computer Interactions
• The Cognitive functions (brainwaves) are used
to move the robot forwards/backwards
• The Expressive functions are used to steer the
robot left/right when the user blinks
accordingly
Initial Evaluation
• An evaluation session has been conducted with
five participants in a laboratory environment
• Feedback was received in direct reply to the
questions, as well as by raising additional issues
• All participants had no previous experience with
BCIs so some time was given to familiarise with
the technology
19/04/2017
7
Initial Evaluation .
• Since all users interacted with a virtual object
using their brain activity for the first time, it was
necessary to perform repeatable profile training
– So players managed to familiarise with the
prototype system
• At this stage, the system extracts and classifies
the player’s intentions more accurately
Initial Evaluation ..
• All participants had to complete
a small task
– 5 to 10 minutes
• The task was to
– Move an avatar inside the Roma
Nova
– Interact with the agents using just
brainwaves and facial expressions
Positive Feedback
• All participants mentioned that it was a
unique experience to interact with the game
through brainwaves
• Even if it was ‘slower’ to interact with the
game they reported that this way of
interaction is far more enjoyable
– Compared to standard input devices such as the
mouse and the keyboard
Positive Feedback .
• All users enjoyed the graphics quality of the
game
– As well as the ‘clever’ dialogues with the intelligent
agents
• The majority of the players mentioned that the
brain computer technologies can be very useful
for interaction in games and it can be combined
with other techniques
Negative Feedback
• Some users found it hard to adapt in taking
control of the agent straight away
– They got distracted by external stimuli
• Some mentioned that it was not easy to
concentrate in the game and they would prefer
a more immersive environment
– Even if through time they started to get control and
adapt to the prototype system
Negative Feedback .
• Finally, some participants found the BCI
technology not as accurate as standard input
devices
– Even if in this particular game there were no
significant requirements on accuracy in navigation,
in other computer games that could be
problematic
19/04/2017
8
End User Evaluation
• 31 users have been evaluated the hybrid BCI
architecture providing feedback by interacting with
the two games
• Each user had to complete a set of tasks to evaluate
efficiently the system and the overall interaction
• EEG data from two mental tasks of the user (push,
pull) had been recorded and stored in order to be
analysed and processed
Cerebral Palsy User Case
• Cerebral palsy (CP) is a motor condition that
causes physical disability in human
development in various areas of body
movement
• A user with Cerebral Palsy had been interacted
successfully with the system, being able to move
the virtual objects despite being affected by
spastic hemiplegia
Results
• 16/31 (51%) users have
reported through their
answers that
they were engaged to the
game
• EEG analysis had been
performed to see if their
answers matches their
brain activity
Results .
• Delta and Theta rhythms (0.1-4-8Hz) are lowfrequency
EEG patterns that increase during
sleep in the normal adult
Results ..
• Beta rhythms (12-30Hz) occur in individuals who
are alert and attentive to external stimuli or
exert specific mental effort
Results …
• 9 out 31 users found
with increased Beta
activity
• That’s 29% of the
users that scored
high on the
engagement related
questions
19/04/2017
9
General Finding
• This proves that user’s experience may is
different from what actually was recorded
through the EEG
– Taking in good faith that the headset measured
accurately
Enobio
BCI illiteracy
Around 20 % of BCI users do not obtain reliable
BCI control (Tan and Nijholt, 2010).
• Investigation of BCI illiteracy can lead to:
• Avoid unnecessary training sessions
• Develop co-adaptive learning strategies to ’cure’
BCI illiteracy
• Understand neurophysiological-basis of BCI
illiteracy
• ... and ultimately build better BCI systems!
Physical factors make
classification difficult.
• Differences in brain anatomy may yield very
variable signal quality
• Large muscle artefacts
(Luck, S. J., 2005)
BCI illiteracy has been shown
in different paradigms.
● Steady-State VEP (SSVEP)
● Evoked Related Potentials (ERP)
● Sensory Motor Rhythms (SMR)
19/04/2017
10
Illiteracy in SSVEP
Illiterate user
Literate user
8 Hz stimulus 13 Hz stimulus
Time Time
SSVEP-BCI illiteracy
Mostly demographic data:
● Age and gender may be important but not significant
● handedness
● vision correction
● tiredness
● time slept
● alcohol
● and caffeine
● computer work
● computer games.. .. had no effect.
(Allison et al., 2010; Volosyak et al., 2011).
Illiteracy in
P300-based BCI
(Tan and Nijholt, 2010)
Literate subject Illiterate subject
Illiteracy in
P300-based BCI
Gender, level of education, working duration and
cigarette and coffee consumption did not show
significant differences. Participants who slept less
than 8 hours showed to perform better (Guger et
al., 2009).
(Halder et al., 2013)
Imagined movements of right and
left limbs may not elicit SMR.
C3 (LH) C4 (RH)C4 (RH)C4 (RH)
Time (sec) Time (sec)
literate subject illiterate subject
Right
hand
(Hz)
Left
hand
(Hz)
Predicting BCI performance
in SMR
● power spectrum density in alpha band in the
(Blankertz et al., 2009)
● motor imagery (Vuckovic, 2010)
● locus of control of reinforcement with (LOC)
regard to dealing with technology (Burde and
Blankertz, 2006)
19/04/2017
11
Predicting BCI performance
in SMR
● performance level (measured by Attitudes
Towards Work test)
● two-hand coordination (Hammer et al., 2012)
● age and daily average amount of hand and
arm movement (Randolph et al., 2006)
● mood and motivation (Nijboer et al., 2008)
Predicting BCI
performance in SMR
Participants with better SMR-BCI
performance employ larger cortical area
during motor imagery. The number of
activated voxels during motor
observation was significantly correlated
with accuracy in the EEG-BCI task
(r=0.53).(Halder et al., 2011)
Two types of attention
• Dorsal, top-down, network (light blue)
• Ventral, bottom-up, network (dark blue)
(Desimone & Duncan, 1995)
Does attention have an effect
on BCI performance?
● Good SMR performers activated prefrontal cortex and
supplementary motor area significantly more than bad performers
which is involved in top-down attention (Halder et al., 2011).
● Attention (measured by block tapping task and digit span) correlated
with ability to control slow cortical potentials (SCP)(Daum et al.,
1993). Control of SCP relies on cortico-basal ganglia-circuits
(Hinterberger et al., 2005).
● Meditation influences BCI performance:
I. Zen meditation improves control of EEG signal (Lo et al., 2004) and
classification accuracy (Eskandari and Erfanian, 2008) .
II. Vipassana and Himalayan meditation resulted in higher peaks frequency
in SSVEP (Karalis et al., 2011; Karalis et al., in press).
III. Mindfulness meditation training improves P300-based BCI performance
(Lakey et al., 2011).
How to combat
BCI illiteracy?
● improve classification accuracy
● change paradigm
● change neuroimaging technique
● combine neuroimaging techniques
● combine paradigms
Effects of attention on
P300-based BCI performance
Three measures:
• Attentional blink
• Posner cueing task
• P300-bic task
19/04/2017
12
Posner cueing task Attentional blink
P300 task
Thank you for your attention!
?
References
• [Allison et al., 2010] Allison, B., Luth, T., Valbuena, D., Teymourian, A., Volosyak, I., and Graser, A. (2010). Bci
demographics: How many (and what kinds of) people can use an ssvep bci? Neural Systems and Rehabilitation
Engineering, IEEE Transactions on, 18(2):107-116.
• [Blankertz et al., 2009] Blankertz, B., Sanelli, C., Halder, S., Hammer, E.-M., Kubler, A., Muller, K.-R., Curio, G., and
Dickhaus, T. (2009). Predicting bci performance to study bci illiteracy.
• [Blankertz et al., 20010] Benjamin Blankertz, Claudia Sannelli, Sebastian Halder, Eva M Hammer, Andrea K•ubler,
Klaus-Robert M•uller, Gabriel Curio, and Thorsten Dickhaus. Neurophysiological predictor of smr-based bci
performance. NeuroImage, 51(4):1303-1309, 2010.
• [Burde and Blankertz, 2006] Burde, W. and Blankertz, B. (2006). Is the locus of control of reinforcement a predictor of
brain-computer interface performance? In Proceedings of the 3rd International Brain-Computer Interface Workshop
and Training Course, volume 2006, pages 108-109.
• [Daum et al., 1993] Daum, I., Rockstroh, B., Birbaumer, N., Elbert, T., Canavan, A., and Lutzenberger, W. (1993).
Behavioural treatment of slow cortical potentials in intractable epilepsy: neuropsychological predictors of outcome.
Journal of Neurology, Neurosurgery & Psychiatry, 56(1):94-97
• [Eskandari and Erfanian, 2008] Eskandari, P. and Erfanian, A. (2008). Improving the performance of brain-computer
interface through meditation practicing. In Engineering in Medicine and Biology Society, 2008. EMBS 2008. 30th
Annual International Conference of the IEEE, pages 662{665. IEEE.
• [Guger et al., 2009] Guger, C., Daban, S., Sellers, E., Holzner, C., Krausz, G., Carabalona, R., Gramatica, F., and
Edlinger, G. (2009). How many people are able to control a p300-based brain-computer interface (bci)? Neuroscience
letters, 462(1):94{98.
• [Halder et al., 2011] Halder, S., Agorastos, D., Veit, R., Hammer, E. M., Lee, S., Varkuti, B., Bogdan, M., Rosenstiel,
W., Birbaumer, N., and K•ubler,A. (2011). Neural mechanisms of brain-computer interface control. Neuroimage,
55(4):1779-1790.
• [Hammer et al., 2012] Hammer, E. M., Halder, S., Blankertz, B., Sannelli, C., Dickhaus, T., Kleih, S., M•uller, K.-R., and
K•ubler,A. (2012). Psychological predictors of smr-bci performance. Biological psychology, 89(1):80-86.
References .
• [Hinterberger et al., 2005] Hinterberger, T., Veit, R., Wilhelm, B., Weiskopf, N., Vatine, J.-J., and Birbaumer, N.
(2005). Neuronal mechanisms underlying control of a brain{computer interface. European Journal of
Neuroscience, 21(11):3169-3181.
• [Karalis et al., ] Karalis, N., Karanasiou, I., and Uzunoglu, N. Short term effects of vipassana meditation in a single
subject ssvep study.
• [Karalis et al., 2011] Karalis, N., Karanasiou, I., Uzunoglu, N., and Braboszcz, C. (2011). Effects of himalayan
tradition meditation during a ssvep study. Neuroscience Letters, 500:e39.
• [Lakey et al., 2011] Lakey, C. E., Berry, D. R., and Sellers, E. W. (2011). Manipulating attention via mindfulness
induction improves p300-based brain-computer interface performance. Journal of neural engineering, 8(2):025019.
• [Lo et al., 2004] Lo, P.-C., Wu, S.-D., and Wu, Y.-C. (2004). Meditation training enhances the efficacy of bci system
control. In Networking, Sensing and Control, 2004 IEEE International Conference on, volume 2, pages 825-828.
IEEE.
• [Luck, 2005] Luck, S. J. (2005). An introduction to the event-related potential technique.
• Nijboer et al., 2008] Nijboer, F., Furdea, A., Gunst, I., Mellinger, J., McFarland, D. J., Birbaumer, N., and K•ubler,A.
(2008). An auditory brain-computer interface (bci). Journal of neuroscience methods, 167(1):43-50.
• [Randolph et al., 2006] Randolph, A., Karmakar, S., and Jackson, M. (2006). Towards predicting control of a braincomputer
interface. ICIS 2006 Proceedings, page 53.
• [Tan and Nijholt, 2010] Tan, D. S. and Nijholt, A. (2010). Brain-Computer Interfaces: applying our minds to humancomputer
interaction. Springer.
• [Volosyak et al., 2011] Volosyak, I., Valbuena, D., Luth, T., Malechka, T., and Graser, A. (2011). Bci demographics
ii: how many (and what kinds of) people can use a high-frequency ssvep bci? Neural Systems and Rehabilitation
Engineering, IEEE Transactions on, 19(3):232{239.
• [Vuckovic, 2010] Vuckovic, A. (2010). Motor imagery questionnaire as a method to detect bci illiteracy. In Applied
Sciences in Biomedical and Communication Technologies (ISABEL), 2010 3rd International Symposium on, pages
1-5. IEEE.