VisioTherapy AI-powered remote physical therapy Petra Budíková https://lh3.googleusercontent.com/DHtVbs3ezJmdKGY_jSUkvqAP9L838SBsx0SU75cP9H_vDD_nryXueWj9d0-OuTXpA aMVuamXKpvKkmgpA5C8K121ppiviCPHF5u8rQYk0ek1MgAwLIICCV-aCJ7v0gSHZpLSUcufxX-o-lTBtDIp2hTKuA=s2048 About me §2013: Ph.D. at FI MUNI (Large-Scale Multi-Modal Image Search) §2013-2021: postdoc at DISA lab, FI MUNI §Image retrieval and annotation §Human motion retrieval §Since 2022: R&D in VisionCraft + little research at FI § §Summer 2023: joined forces with DISA lab within TACR grant §Objective: develop VisioTherapy software for remote monitoring of rehabiliation exercising § 2/18 About VisionCraft §Startup from Brno, founded in 2018 §At first, focus on Intelligent Transportation Systems §Object detection and tracking using NNs §Edge computing, privacy-safe §Since 2022, another area of interest: applications of human motion analysis for healthcare https://uploads-ssl.webflow.com/6422ecf23df0a76dd30ae088/649eb65f1ee9ae72203ef833_team.jpg 3/18 Human motion analysis for dummies What is the man doing? Is his movement similar to some example? Is he dangerous? … Video taken by standard camera/webcam/ smartphone 3D model of human skeleton motion data extraction motion data analysis 4/18 Why physical therapy? 5/18 Why physical therapy? (II) §Home exercising issues §50-70% of patients do not adhere to home exercise plans §Patients often exercise incorrectly or do not even remember the exercise §Poor accessibility §Waiting times about 9-15 weeks to see a physical therapist is US and UK §In USA, it is estimated that 50 000 therapists will be missing by 2050 §Enormous demand §A third of people globally are currently living with a health condition that would benefit from rehabilitation §The number is expected to grow in future § 6/18 VisioTherapy concept §Basic idea: Assist patients during home exercising §Guide them during exercising §Detect errors, provide feedback §Provide motivation, measure progress §Report to therapists § §Patient’s journey: § Initial appointment & individual- isation Exercising at home guided by VisioTherapy app (Ir)regular checks with therapist 7/18 VisioTherapy concept: UI + acoustic feedback (sound signals or synthetic voice) https://lh3.googleusercontent.com/DHtVbs3ezJmdKGY_jSUkvqAP9L838SBsx0SU75cP9H_vDD_nryXueWj9d0-OuTXpA aMVuamXKpvKkmgpA5C8K121ppiviCPHF5u8rQYk0ek1MgAwLIICCV-aCJ7v0gSHZpLSUcufxX-o-lTBtDIp2hTKuA=s2048 + acoustic feedback (sound signals or synthetic voice) 8/18 VisioTherapy concept: Benefits §For patients § § § § § §For therapists § vreminders to exercise vmotivating elements vcontrol of correct exercising frequent and correct exercising faster recovery vmotivated patients vfeedback from patient’s home exercising more efficient patient treatment more money 9/18 VisioTherapy concept: What makes us special §Wide availability: runs on smartphone, no special hardware needed §Absolute privacy: edge computing, no video shared anywhere §Fast and precise feedback: real-time motion analysis, high-precision motion detection §Personalized solution: takes into account the individual needs and abilities of each patient §Therapist in the loop: therapist remains in charge, guarantees the medical quality of the rehabilitation 10/18 VisioTherapy: first steps §Autumn 2022: first experiments and consultations with therapists §Very rough prototype presented at WebSummit conference, positive feedback §January 2023: submission of TACR TREND grant proposal §Scope: in cooperation with FI, develop more precise algorithms for skeleton detection and motion analysis, create working desktop application §May 2023: submission of TACR SIGMA grant proposal §Scope: develop of mobile application, gamification, business strategy §Summer 2023: received both grants, can build a reasonable team and start serious work on VisioTherapy!! §Developers and business people: VisionCraft §Physiotherapy expert: Mgr. Lukáš Katzer §Research team: Honza Sedmidubský, Mima Jánošová, Andrej Černek, Dávid Rusnák (DP) 11/18 VisioTherapy: from idea to product §Extraction of high-quality motion data from video §Precise and fast evaluation of exercise correctness §Analysis of patient’s progress, problematic parts, … §Gamification elements §Design and implementation of desktop and mobile app § https://lh3.googleusercontent.com/CdP5s8eav3mtmau0v-p4JEDbm2OMZ_1ljge5QaE9_YcMpKIjHzKk_7IryZISlD1IF i6ECjZPQUOhSngrky2U3c5Y9ZU8TTBNHaNoDHL-A8htfGrWaW2uHBg85V43iHlBPIj7b0a_e2KpSb7MlrSvnFFj6g=s2048 12/18 C:\Users\Admin\Desktop\unicorn.jpg Extraction of high-quality motion data from video §On one hand, there are many NNs that perform human pose extraction from ordinary video §On the other hand, the results are often far from perfect § §What can we do to extract high-quality motion data? §Search for NN models suitable for rehabilitation exercises §Create dataset of rehabilitation exercises with GT of precise motion data §HCI lab at FI §Fine-tune existing NN models using our data §Preprocess video before applying NNs §Apply cleaning algorithms to extracted motion data § C:\Users\Admin\Downloads\20230906_100415.jpg 13/18 Motion analysis §Basic functionality §Checking static constraints on patient’s position §Measurement of joint movement angles §Advanced processing of motion data §Definition of suitable similarity model §Online detection of erroneous exercising §Offline comparison of motion sequences using both positive and negative examples §Exercise repetition counting §Detection and explanation of differences between model and actual exercise §Patient progress monitoring §Data mining over exercising data of patient groups 14/18 Gamification §Many apps use gamification §Language learning §Fitness training §… §However, not all gamification principles are suitable for physiotherapy §The more the better is not true here! §We need to consult with therapists to come up with suitable motivation elements §Could be a nice bachelor/diploma theses… anybody interested? 15/18 Design and implementation of VisioTherapy app §Both desktop and mobile interfaces §Mobile for patients §Mobile and desktop for therapists §Challenge: Feedback on exercise correctness §During exercising, after exercising §Visual and audio §This could also be a very nice bachelor/diploma thesis! §Implementation §Edge computing – emphasis on efficiency §Only derived data (MoCap + statistics) sent to server § 16/18 Timeline https://lh3.googleusercontent.com/CdP5s8eav3mtmau0v-p4JEDbm2OMZ_1ljge5QaE9_YcMpKIjHzKk_7IryZISlD1IF i6ECjZPQUOhSngrky2U3c5Y9ZU8TTBNHaNoDHL-A8htfGrWaW2uHBg85V43iHlBPIj7b0a_e2KpSb7MlrSvnFFj6g=s2048 7/2023 Very rough prototype Basic motion analysis 7/2024 Full prototype for desktop Enhanced quality of motion data Evaluation of exercise correctness Exercise repetition counting Ready to test with first users 7/2025 Working mobile app High-quality motion data Full scope of motion analysis 17/18 C:\Users\Admin\Desktop\unicorn.jpg The End… Questions? Comments? feedback, insights, socialize 18/18