Semantically Consistent Human Motion Segmentation Michal Balazia Jan Sedmidubsky Pavel Zezula Laboratory of Data Intensive Systems and Applications Faculty of Informatics Masaryk University March 24, 2014 Michal Balazia (DISA) Human Motion Segmentation March 24, 2014 1 / 14 Presentation Outline O 3D Human Motion Capture and Representation O Key poses Q Existing approaches to key-pose detection O Our purpose of motion segmentation O How our method works O Results Michal Baläzia (DISA) Human Motion Segmentation March 24, 2014 2 / 14 3D Human Motion Capture and Representation Employment • Motion simulation and exposition - avatar graphics a Content-based retrieval in motion databases • Biomechanical analysis - detection of gait disorders, rehabilitation Generating new motion instances Neck Right \ B.Hand Shoulder •-•- Lett Shoulder L Hand —• KOCH • 3D data capturable via Microsoft Kinect or on-body sensors • Human motion in the form of a sequence of body poses a Pose characterised by 3D coordinates of selected body points and time • Various extractable features • Joint angles, angular velocity, acceleration • Joint relative distances • Relational features 1 -00.0 Michal Baläzia (DISA) Human Motion Segmentation March 24, 2014 3 / 14 Key poses • Not all poses are important for us • We detect key poses - the poses of interest • Key pose selection according to the purpose • Purposes • Summarization and compression of motion data • Motion retrieval • Motion sequence segmentation • Relevant approaches to key-pose detection - Assa, Gong, Xiao Michal Balazia (DISA) Human Motion Segmentation March 24, 2014 4 / 14 Assa Assa - based on curve averaging • Pose described skeletal joints and their associated aspects: (1) positions, (2) angles, (3) velocity, (4) angular velocity • ft - value of aspect a at time t • High-dimensional curve (4x^joints) is reduced by RMDS algorithm to a curve C(t) of 5-8 dimensions • Point t in C(t) is projected onto a predefined average curve C(t) • rt = \C(t) — C(t)\ - distance at time t • Iterative key-pose detection algorithm: 1. Find pose index t of maximum rt 2. Mark the t-th pose as key pose 3. Modify C(t) to touch C(t) Michal Balazia (DISA) Human Motion Segmentation March 24, 2014 5 / 14 Gong Gong - based on local-motion energy extremes • Human model consists of 12 limb bones • Motion features in form of limb bone axis rotations • 6lt,a - angle between limb bone I and axis a at time t 9 36-dimensional pose Pt: \cos(6\,x), cos(9l,y), cos{0].,z),..., cos{9]:2,x), cos{9]2,y), cos{9]2,z)] • Local-motion energy in pose at time t Et = \Pt - Pt-i\2 • Key poses are the poses Pt of locally extremal Et Michal Balážia (DISA) Human Motion Segmentation March 24, 2014 6 / 14 Xiao Xiao - based on curve simplification • Human model consists of 8 limb bones • Motion features in form of limb bone angles • 0\ - angle between limb bone I and central bone at time t • 8-dimensional pose Pt: $,...,0?] • Key-pose candidates are the poses where any Q\ is locally extremal • Curve is constructed to enclose the key-pose candidates • LCS (similar to SCS) is applied to this curve to refine key poses Michal Balazia (DISA) Human Motion Segmentation March 24, 2014 7 / 14 Our Purpose of Motion Segmentation Our purpose: motion retrieval • Pose-level indexing - inefficient • Segment-level indexing • Both motion database and query are segmented • Segments can be separately analysed, indexed, clustered, or compared • Query is searched in the database as a sequence of motion segments • Semantic consistency - key poses at semantically equivalent phases i i i i i i i i ii i i i i i i i Poses Poses (a) (b) Michal Balazia (disa) Human Motion Segmentation March 24, 2014 8 / 14 How Our Method Works Our - based on changes in body point trajectories • Database: HDM05 Motion Capture Database - University of Bonn • Two major steps: 1. Feature extraction 2. Pose energy curve construction and analysis 1. Feature extraction • Relative distances of selected body point pairs: 1. left hand and right hand 2. left hand and left foot 3. right hand and right foot 4. left knee and right knee 5. left foot and right foot • Motion - sequence of poses M. = (Vi,..., Vn) 9 Pose - configuration of features in time Vt = (ft, • • • > ft) • f? - relative distance of p-th body point pair at time t Michal Balazia (DISA) Human Motion Segmentation March 24, 2014 9 / 14 How Our Method Works 2. Pose energy curve construction and analysis • Energy is determined by pose neighbourhood of radius e • Feature energy of each feature ff at pose Pt is calculated as E feature i.f\ t+e E t'=t-e (ff - m • Pose energy of each pose Pt is calculated as Epose(Pt) — ^ ^ \Efeature(ft p=l i/lotion Segmentation How Our Method Works • Pose-energy curve is constructed by concatenating the pose-energy values through the processed motion in chronological order i • Local maxima greater than a r threshold of this curve are identified as key poses i/lotion Segmentation Results We have: • Extracted body point distances as motion features • Manually defined ground truth over 11 motion sequences • Implemented and tested all methods - our, Assa, Gong, Xiao Evaluation results: • Evaluation against ground truth: hit within 10-pose neighbourhood Recall (%) Precision (%) F-measure (%) Ratio (%) Our 52.2 46.5 49.2 112.1 Assa 35.2 39.2 37.1 89.7 Gong 92.1 4.6 8.7 2040.8 Xiao 30.7 45.3 36.6 67.7 i-0 0.0 i/lotion Segmentation Results • Evaluation of semantic consistency of all methods 100 60 80 100 120 140 160 180 200 DTW distance threshold ■00.0 notion Segmentation Questions? Thank you for attention. J. Assa, Y. Caspi, and D. Cohen-Or, Action Synopsis: Pose Selection and Illustration, ACM SIGGRAPH 2005. W. Gong, A.D. Bagdanov, F.X. Roca, J. Gonzalez, Automatic Key Pose Selection for 3D Human Action Recognition, AMDO 2010. J. Xiao, Y. Zhuang, T. Yang, and F. Wu, An Efficient Keyframe Extraction from Motion Capture Data, CGI 2006. Michal Baläzia (DISA) Human Motion Segmentation March 24, 2014 14 / 14