You Are How You Walk:
Gait Recognition from Motion Capture Data
Michal Balazia
Faculty of Informatics, Masaryk University, Brno, Czech Republic
https://gait.fi.muni.cz
Michal Balazia (FI MU) Gait Recognition from MoCap Data May 2nd, 2017 1 / 26
Human Identification by Gait
Data captured by a system of multiple cameras or a depth camera
Large tracking space
Multiple samples of a single walker
High variance in encounter conditions
Database of large amount of biometric samples
Identification in real time
Michal Balazia (FI MU) Gait Recognition from MoCap Data May 2nd, 2017 2 / 26
Motion Capture Data (MoCap)
Frame: 0 25 50 75 100 125 150
y
x
z
Structural motion data
Skeleton of joints and bones
Data = 3D positions of joints in time.
Can be collected by a system of multiple cameras (Vicon) or a depth
camera (Microsoft Kinect)
Michal Balazia (FI MU) Gait Recognition from MoCap Data May 2nd, 2017 3 / 26
Raw MoCap Gait Data
Model of human body has J joints
Measured gait cycle has length of T video frames
Raw MoCap gait sample is a tensor
g =



γ1 (1) · · · γ1 (T)
...
...
...
γJ (T) · · · γJ (T)



γj (t) ∈ R3 are 3D coordinates of j ∈ {1, . . . , J} at t ∈ {1, . . . , T}
Dimensionality 3JT
Sample space {g}
Michal Balazia (FI MU) Gait Recognition from MoCap Data May 2nd, 2017 4 / 26
Geometric Features
Examples of geometric gait features:
joint angles (angle in shoulder-elbow-wrist)
inter-joint distances (feet distance)
joint velocity or acceleration
areas of joint polygons (upper body span)
. . .
feature signals
video frames
features
video frames
Examples of distance functions:
Dynamic Time Warping
Minkowski distances
. . .
Michal Balazia (FI MU) Gait Recognition from MoCap Data May 2nd, 2017 5 / 26
Linearly Learned Latent Features
Labeled learning sample space {(gn, n)}NL
n=1
n is a label of one of the identity classes {Ic}CL
c=1
Ic has a priori probability pc
Consider an optimization criterion J
Feature extraction is given by a feature matrix Φ ∈ RD×D
D-dimensional sample space {gn}N
n=1
D-dimensional feature space {gn}N
n=1
Transform gait samples gn into gait templates gn = Φ gn
Examples of distance functions:
Mahalanobis distance
Minkowski distances
. . .
Michal Balazia (FI MU) Gait Recognition from MoCap Data May 2nd, 2017 6 / 26
Learning by MMC
Optimize class separability of the feature space
Margin of two classes is the Euclidean distance of their means µc
minus both their variances Σc
Maximum Margin Criterion used by the Support Vector Machines
J =
1
2
CL
c,c =1
pcpc (µc − µc ) (µc − µc ) − tr (Σc + Σc )
= . . . = tr (ΣB − ΣW)
Between-class scatter matrix ΣB, within-class scatter matrix ΣW
Criterion for a feature matrix Φ
J (Φ) = tr Φ (ΣB − ΣW) Φ
Solution: solve the generalized eigenvalue problem
(ΣB − ΣW) Φ = ΛΦ
Mahalanobis distance function on templates
Michal Balazia (FI MU) Gait Recognition from MoCap Data May 2nd, 2017 7 / 26
Learning by PCA+LDA
2-stage feature extraction technique
Principal Component Analysis and Linear Discriminant Analysis
Total scatter matrix ΣT = ΣB + ΣW
Criterion for a feature matrix ΦLDA
J (ΦPCA) = tr ΦPCAΣTΦPCA
J (ΦLDA) = tr
ΦLDAΦPCAΣBΦPCAΦLDA
ΦLDAΦPCAΣWΦPCAΦLDA
Solution: solve the generalized eigenvalue problems
ΣTΦPCA = ΛΦPCA
ΦPCAΣWΦPCA
−1
ΦPCAΣBΦPCA ΦLDA = ΛΦLDA
Mahalanobis distance function on templates
Michal Balazia (FI MU) Gait Recognition from MoCap Data May 2nd, 2017 8 / 26
Identity Classification Pipeline
Spo�ed walker
Acquiring
MoCap data
Phase I
Detec�ng
gait
Phase II
Iden�fying
walkers
Phase IV
MoCap data
Gait template
Gait sample
Iden�fied walker
DEPARTURES
BANK
OPENI
NG HOUR
S
BANK
Extrac�ng
gait features
Phase III
Michal Balazia (FI MU) Gait Recognition from MoCap Data May 2nd, 2017 9 / 26
Identity Classification Pipeline
39° 43' 04'' N 104° 51' 50'' W
2017/06/24 22:49:38
39° 49' 06'' N 104° 52' 10'' W
2017/06/23 13:24:19
39° 51' 05'' N 104° 40' 34'' W
2017/06/21 07:55:16
DEPARTURES
BANK
Michal Balazia (FI MU) Gait Recognition from MoCap Data May 2nd, 2017 10 / 26
The Classification Problem
Identity: Label of a registered identity class.
Michal Balazia (FI MU) Gait Recognition from MoCap Data May 2nd, 2017 11 / 26
But In Video Surveillance Environment ...
How can we represent the identity in video surveillance environment?
Michal Balazia (FI MU) Gait Recognition from MoCap Data May 2nd, 2017 12 / 26
The Class Discovery Problem
Identity: Content of a discovered identity class = movement history.
You are how you walk.
time
crime present
Michal Balazia (FI MU) Gait Recognition from MoCap Data May 2nd, 2017 13 / 26
Universal Gait Features
Data need to be acquired without walker’s consent
New identities can appear on the fly
Labels for all encountered people may not always be available
Universal gait features – features of a high power in recognizing all
people and not only those they were learned on
We learn the universal gait features by MMC or by PCA+LDA on an
auxiliary dataset
The dataset is assumed to be rich on covariate conditions – aspects
of walk people differ in
These features create an unsupervised environment particularly
suitable for uncooperative person identification
Michal Balazia (FI MU) Gait Recognition from MoCap Data May 2nd, 2017 14 / 26
Evaluation: Database
ASF/AMC format of MoCap data
CMU MoCap database obtained from the CMU Graphics Lab
Extracted database contains only gait cycles (motions of two steps)
Normalization: position, walk direction and skeleton
7 extracted databases:
# identities # gait cycles
2 35
4 67
8 130
16 302
32 2,047
54 3,843
64 5,923
Michal Balazia (FI MU) Gait Recognition from MoCap Data May 2nd, 2017 15 / 26
Evaluation: Data Separation
Data separation
Identities Samples
Normal evaluation
EvaluationLearning
Identities
Samples
Cross-identity evaluation
Evaluation
Learning
Evaluation of classification estimated by nested cross-validation
Michal Balazia (FI MU) Gait Recognition from MoCap Data May 2nd, 2017 16 / 26
Evaluation: Metrics
Class separability coefficients:
Davies-Bouldin Index (DBI)
Dunn Index (DI)
Silhouette Coefficient (SC)
Fisher’s Discriminant Ratio (FDR)
Classification based metrics:
Cumulative Match Characteristic
False Accept / Reject Rate
Receiver Operating Characteristic (ROC)
Recall / Precision Rate
Correct Classification Rate (CCR)
Equal Error Rate (EER)
Area Under ROC Curve (AUC)
Mean Average Precision (MAP)
Michal Balazia (FI MU) Gait Recognition from MoCap Data May 2nd, 2017 17 / 26
Evaluation: Results
class separability coefficients classification based metrics scalability
method DBI DI SC FDR CCR EER AUC MAP DCT TD
Ahmed 216.2 0.842 −0.246 0.954 0.657 0.38 0.659 0.165 0.01 24
Ali 501.5 0.26 −0.463 1.175 0.225 0.384 0.679 0.111 0.01 2
Andersson 142.3 1.297 −0.102 1.127 0.84 0.343 0.715 0.251 0.01 68
Ball 161 1.458 −0.163 1.117 0.75 0.346 0.711 0.231 0.01 18
Dikovski 144.5 1.817 −0.135 1.227 0.881 0.363 0.695 0.254 0.01 71
Gavrilova 185.8 1.708 −0.164 0.77 0.891 0.374 0.677 0.254 44.78 5,254
Jiang 206.6 1.802 −0.249 0.85 0.811 0.395 0.657 0.242 8.17 584
Krzeszowski 154.1 1.982 −0.147 0.874 0.915 0.392 0.662 0.275 35.32 3,795
Kumar 118.6 1.618 −0.086 1.09 0.801 0.459 0.631 0.217 7.87 13,950
Kwolek 150.9 1.348 −0.084 1.175 0.896 0.358 0.723 0.323 0.06 660
Preis 1,980.6 0.055 −0.512 1.067 0.143 0.401 0.626 0.067 0.01 13
Sedmidubsky 398.1 1.35 −0.425 0.811 0.543 0.388 0.657 0.149 5.79 292
Sinha 214.8 1.112 −0.215 1.101 0.674 0.356 0.697 0.191 0.01 45
MMCBR 154.2 1.638 0.062 1.173 0.925 0.297 0.748 0.353 0.01 53
MMCJC 130.3 1.891 0.051 1.106 0.918 0.378 0.721 0.315 0.01 51
PCALDABR 182 1.596 −0.015 0.984 0.918 0.361 0.695 0.276 0.01 54
PCALDAJC 174.4 1.309 −0.091 0.827 0.863 0.44 0.643 0.201 0.01 54
Random 0.042 0
RawBR 163.7 2.092 0.011 0.948 0.966 0.315 0.743 0.358 70.27 8,229
RawJC 155.1 1.954 −0.12 0.897 0.926 0.377 0.679 0.283 160.64 13,574
Michal Balazia (FI MU) Gait Recognition from MoCap Data May 2nd, 2017 18 / 26
Evaluation: Results
Michal Balazia (FI MU) Gait Recognition from MoCap Data May 2nd, 2017 19 / 26
Evaluation: Results
Homogeneous set-up with CL = CE ∈ {2, . . . , 27}
Heterogeneous set-up with CL = CE ∈ {2, . . . , 27}
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
21
28
35
42
49
56
63
70
DBI homogeneous
DBI heterogeneous
CCR homogenous
CCR heterogeneous
0.0
0.1
0.2
0.3
0
7
14
21
(2,2)
(3,3)
(4,4)
(5,5)
(6,6)
(7,7)
(8,8)
(9,9)
(10,10)
(11,11)
(12,12)
(13,13)
(14,14)
(15,15)
(16,16)
(17,17)
(18,18)
(19,19)
(20,20)
(21,21)
(22,22)
(23,23)
(24,24)
(25,25)
(26,26)
(27,27)
CCR heterogeneous
40%
60%
80%
100%
120%
DBI
CCR
0%
20%
40%
(2,2)
(3,3)
(4,4)
(5,5)
(6,6)
(7,7)
(8,8)
(9,9)
(10,10)
(11,11)
(12,12)
(13,13)
(14,14)
(15,15)
(16,16)
(17,17)
(18,18)
(19,19)
(20,20)
(21,21)
(22,22)
(23,23)
(24,24)
(25,25)
(26,26)
(27,27)
CCR
Michal Balazia (FI MU) Gait Recognition from MoCap Data May 2nd, 2017 20 / 26
Evaluation: Results
Heterogeneous set-up with CL ∈ {2, . . . , 27} and CE = 27
Heterogeneous set-up with CL ∈ {2, . . . , 52} and CE = 54 − CL
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
300
400
500
600
700
800
0.0
0.1
0.2
0.3
0
100
200
(2,27)
(3,27)
(4,27)
(5,27)
(6,27)
(7,27)
(8,27)
(9,27)
(10,27)
(11,27)
(12,27)
(13,27)
(14,27)
(15,27)
(16,27)
(17,27)
(18,27)
(19,27)
(20,27)
(21,27)
(22,27)
(23,27)
(24,27)
(25,27)
(26,27)
(27,27)
DBI
CCR
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
600
800
1000
1200
1400
1600
1800
2000
0.0
0.1
0.2
0.3
0
200
400
600
(2,52)
(3,51)
(4,50)
(5,49)
(6,48)
(7,47)
(8,46)
(9,45)
(10,44)
(11,43)
(12,42)
(13,41)
(14,40)
(15,39)
(16,38)
(17,37)
(18,36)
(19,35)
(20,34)
(21,33)
(22,32)
(23,31)
(24,30)
(25,29)
(26,28)
(27,27)
(28,26)
(29,25)
(30,24)
(31,23)
(32,22)
(33,21)
(34,20)
(35,19)
(36,18)
(37,17)
(38,16)
(39,15)
(40,14)
(41,13)
(42,12)
(43,11)
(44,10)
(45,9)
(46,8)
(47,7)
(48,6)
(49,5)
(50,4)
(51,3)
(52,2)
Michal Balazia (FI MU) Gait Recognition from MoCap Data May 2nd, 2017 21 / 26
Evaluation: Results
Table 2. Classification metrics for all implemented methods. Homogeneous setup, evaluated on the full
database with 10-fold cross-validation. Methods are ordered by their CCR score.
method CCR EER AUC MAP CMC
RawJC 0.872 0.321 0.731 0.317
0.45
0.50
0.55
0.60
0.65
0.70
0.75
0.80
0.85
0.90
0.95
1.00 Raw(JC)
MMC(BR)
Raw(BR)
MMC(JC)
PCA+LDA(BR)
KwolekB
KrzeszowskiT
PCA+LDA(JC)
DikovskiB
AhmedF
AnderssonVO
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
12345678910
AnderssonVO
NareshKumarMS
JiangS
BallA
SinhaA
AhmedM
SedmidubskyJ
AliS
PreisJ
Random
0.45
0.50
0.55
0.60
0.65
0.70
0.75
0.80
0.85
0.90
0.95
1.00
CumulativeCorrectClassificationRate
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
1 2 3 4 5 6 7 8 9 10
CumulativeCorrectClassificationRate
Rank
MMCBR 0.868 0.305 0.739 0.332
RawBR 0.867 0.333 0.701 0.259
MMCJC 0.861 0.325 0.72 0.309
PCA+LDABR 0.845 0.335 0.682 0.247
KwolekB 0.823 0.367 0.711 0.296
KrzeszowskiT 0.802 0.348 0.717 0.273
PCA+LDAJC 0.79 0.417 0.634 0.189
DikovskiB 0.787 0.376 0.679 0.227
AhmedF 0.771 0.371 0.664 0.22
AnderssonVO 0.76 0.352 0.703 0.228
NareshKumarMS 0.717 0.459 0.613 0.19
JiangS 0.692 0.407 0.637 0.204
BallA 0.667 0.356 0.698 0.207
SinhaA 0.598 0.362 0.69 0.176
AhmedM 0.58 0.392 0.646 0.145
SedmidubskyJ 0.464 0.394 0.65 0.138
AliS 0.186 0.394 0.662 0.096
PreisJ 0.131 0.407 0.618 0.066
Random 0.039
Michal Balazia (FI MU) Gait Recognition from MoCap Data May 2nd, 2017 22 / 26
Evaluation: Results
48
48
48
48
49
49
49
49
49
49
49
49
49
49
50
50
50
50
50
50
50
50
50
50
51
51
51
51
B IJCB
CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.
0
0.5
DCT Ahmed Andersson Dikovski Preis MMCAhmed Andersson Dikovski Preis MMC
Ali Ball Kwolek Sinha PCA+LDA
10
100
DBI
Ahmed
Ali
Andersson
Ball
Dikovski
Kwolek
1
(2,62)
(3,61)
(4,60)
(5,59)
(6,58)
(7,57)
(8,56)
(9,55)
(10,54)
(11,53)
(12,52)
(13,51)
(14,50)
(15,49)
(16,48)
(17,47)
(18,46)
(19,45)
(20,44)
(21,43)
(22,42)
(23,41)
(24,40)
(25,39)
(26,38)
(27,37)
(28,36)
(29,35)
(30,34)
(31,33)
(32,32)
Kwolek
Preis
Sinha
MMC
PCA+LDA
(a) DBI
-0.4
-0.3
-0.2
-0.1
0.0
0.1
SC
Ahmed
Ali
Andersson
Ball
Dikovski
Kwolek
-0.6
-0.5
-0.4
(2,62)
(3,61)
(4,60)
(5,59)
(6,58)
(7,57)
(8,56)
(9,55)
(10,54)
(11,53)
(12,52)
(13,51)
(14,50)
(15,49)
(16,48)
(17,47)
(18,46)
(19,45)
(20,44)
(21,43)
(22,42)
(23,41)
(24,40)
(25,39)
(26,38)
(27,37)
(28,36)
(29,35)
(30,34)
(31,33)
(32,32)
Kwolek
Preis
Sinha
MMC
PCA+LDA
(b) SC
0.65
0.70
0.75
0.80
ROC
Ahmed
Ali
Andersson
Ball
Dikovski
Kwolek
0.55
0.60
(2,62)
(3,61)
(4,60)
(5,59)
(6,58)
(7,57)
(8,56)
(9,55)
(10,54)
(11,53)
(12,52)
(13,51)
(14,50)
(15,49)
(16,48)
(17,47)
(18,46)
(19,45)
(20,44)
(21,43)
(22,42)
(23,41)
(24,40)
(25,39)
(26,38)
(27,37)
(28,36)
(29,35)
(30,34)
(31,33)
(32,32)
Kwolek
Preis
Sinha
MMC
PCA+LDA
(c) AUC
0.15
0.20
0.25
0.30
0.35
0.40
PR
Ahmed
Ali
Andersson
Ball
Dikovski
Kwolek
0.00
0.05
0.10
(2,62)
(3,61)
(4,60)
(5,59)
(6,58)
(7,57)
(8,56)
(9,55)
(10,54)
(11,53)
(12,52)
(13,51)
(14,50)
(15,49)
(16,48)
(17,47)
(18,46)
(19,45)
(20,44)
(21,43)
(22,42)
(23,41)
(24,40)
(25,39)
(26,38)
(27,37)
(28,36)
(29,35)
(30,34)
(31,33)
(32,32)
Kwolek
Preis
Sinha
MMC
PCA+LDA
(d) MAP
Figure 3: Simulations with 31 different (CL,CE) configurations (horizontal axes) on four evaluation metrics (vertical axes).
100
105
110
DBIMichal Balazia (FI MU) Gait Recognition from MoCap Data May 2nd, 2017 23 / 26
Evaluation: Results
70
75
80
85
90
95
100
105
110
%score
DBI
SC
ROC
PR
65
70
root
lhipjoint
rhipjoint
lfemur
rfemur
ltibia
rtibia
lfoot
rfoot
ltoes
rtoes
lclavicle
rclavicle
lhumerus
rhumerus
lradius
rradius
lwrist
rwrist
lhand
rhand
lfingers
rfingers
lthumb
rthumb
lowerback
upperback
thorax
lowerneck
upperneck
head
pelvis
torso
leftarm
rightarm
arms
leftleg
rightleg
legs
leftlimbs
rightlimbs
limbs
allbuttorso
allbutarms
allbutlegs
excluded
0
0.5
DCT
Ahmed Andersson Dikovski Preis MMCAhmed Andersson Dikovski Preis MMC
Ali Ball Kwolek Sinha PCA+LDA
(
(
(
(
(
(
(
(
(
(
(
(
(
(
(
(
(
(
(
(
(
(
(
(c) AUC
(
(
(
(
(
(
(
(
(
(
(
(
(
(
(
(
(
(
(
(
(
(
(
(d) MAP
Figure 3: Simulations with 31 different (CL,CE) configurations (horizontal axes) on four evaluation metrics (vertical axes).
70
75
80
85
90
95
100
105
110
%score
DBI
SC
ROC
PR
65
70
root
lhipjoint
rhipjoint
lfemur
rfemur
ltibia
rtibia
lfoot
rfoot
ltoes
rtoes
lclavicle
rclavicle
lhumerus
rhumerus
lradius
rradius
lwrist
rwrist
lhand
rhand
lfingers
rfingers
lthumb
rthumb
lowerback
upperback
thorax
lowerneck
upperneck
head
pelvis
torso
leftarm
rightarm
arms
leftleg
rightleg
legs
leftlimbs
rightlimbs
limbs
allbuttorso
allbutarms
allbutlegs
excluded
Figure 4: Four evaluated metrics of the MMC method with incomplete data. In each column a subset of joints is systematically
excluded from the input: first 31 columns (root – head) exclude a single joint and last 14 columns (pelvis – all but legs)
exclude multiple joints. Structure of the human body is the following: head, pelvis = {root, lhipjoint, rhipjoint},
left leg = {lfemur, ltibia, lfoot, ltoes}, left arm = {lhumerus, lradius, lwrist, lhand, lfingers, lthumb},
right leg = {rfemur, rtibia, rfoot, rtoes}, right arm = {rhumerus, rradius, rwrist, rhand, rfingers, rthumb},
torso = {lowerback, upperback, thorax, lowerneck, upperneck, lclavicle, rclavicle}. Configuration (9, 55).
0.60
0.65
0.70
0.75
0.80
ROC
Ahmed
Ali
Andersson
Ball
Dikovski
Kwolek
Preis
0.50
0.55
0 10 20 30 40 50 60 70 80 90 100
% noise
Preis
Sinha
MMC
PCA+LDA
0.10
0.15
0.20
0.25
0.30
PR
Ahmed
Ali
Andersson
Ball
Dikovski
Kwolek
Preis
0.00
0.05
0 10 20 30 40 50 60 70 80 90 100
% noise
Preis
Sinha
MMC
PCA+LDA
(a) ROC and PR for an x% noise simulated by multiplying each
measured value by a random number in interval (1 − x/100, 1 + x/100).
0.60
0.65
0.70
0.75
0.80
ROC
Ahm
Ali
Ande
Ball
Diko
Kwol
Preis
0.50
0.55
0 10 20 30 40 50 60 70 80 90 100
% noise
Preis
Sinha
MMC
PCA+
0.10
0.15
0.20
0.25
0.30
PR
0.00
0.05
0 10 20 30 40 50 60 70 80 90 100
% noise
(b) ROC and PR for an x% noise simulated by substituting each
measured value with a random value with probability x%.
Figure 5: ROC and PR of all methods with noisy data. Configuration (9, 55).
Michal Balazia (FI MU) Gait Recognition from MoCap Data May 2nd, 2017 24 / 26
Evaluation Framework and Database
Available online at https://gait.fi.muni.cz
Database extraction drive
Implementations of all 20 methods
Classifier learning and classification mechanism
Evaluation mechanism and 12 performance metrics
Michal Balazia (FI MU) Gait Recognition from MoCap Data May 2nd, 2017 25 / 26
Summary
Universal gait features learned on an auxiliary dataset
Our approach based on MMC and PCA+LDA
Broad evaluation on normal and cross-identity setups
MMC learned on 17 identities recognizes 37 identities with 95% CCR
MMC learned yet on 7 identities best recognizes 57 identities
Evaluation framework and database
https://gait.fi.muni.cz
Thank you for attention.
Questions?
Michal Balazia (FI MU) Gait Recognition from MoCap Data May 2nd, 2017 26 / 26