Skip Navigation Links.
American Journal of Sports Science and Medicine. 2020, 8(1), 23-35
DOI: 10.12691/AJSSM-8-1-5
Original Research

OpenButterfly: Multimodal Rehabilitation Analysis of Immersive Virtual Reality for Physical Therapy

Michael Ora Powell1, , Aviv Elor2, Mircea Teodorescu1 and Sri Kurniawan2

1Department of Electrical and Computer Engineering, University of California - Santa Cruz, Santa Cruz, CA, USA

2Department of Computational Media, University of California - Santa Cruz, Santa Cruz, CA, USA

Pub. Date: June 23, 2020
Full Text PDF

Cite this paper

Michael Ora Powell, Aviv Elor, Mircea Teodorescu and Sri Kurniawan. OpenButterfly: Multimodal Rehabilitation Analysis of Immersive Virtual Reality for Physical Therapy. American Journal of Sports Science and Medicine. 2020; 8(1):23-35. doi: 10.12691/AJSSM-8-1-5

Abstract

Upper limb injury often requires repetitive and long-term physical rehabilitation which can result in low adherence due to the repetitive and internally motivated nature of the exercises. Immersive Virtual Reality (iVR) systems enhanced with games can address these challenges. These systems provide a platform for adaptable sensing and analytical tools to track progress, personalize therapy, and increase long term engagement. This paper explores such a system, through an iVR-based experience for upper-extremity rehabilitation called “OpenButterfly,” where users follow movements to protect a virtual butterfly. OpenButterfly enables a dynamically controllable environment for individual exercise by utilizing motion capture, a biomechanical model of torque and angular momentum, and a biometric pipeline for brainwave, heartrate, and skin conductance analysis. We examine this experience for five adult users with varying degrees of injury over the course of eight weeks. Our results suggest that experiences like OpenButterfly provide strong platforms for long-term physical therapy engagement, analysis, and recovery. Lastly, this paper concludes with considerations for future research into adaptive iVR physio-rehabilitation.

Keywords

virtual reality, biofeedback, biomechanical simulation, OpenSim, rehabilitation, exergame, serious games

Copyright

Creative CommonsThis work is licensed under a Creative Commons Attribution 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

References

[1]  C. H. Linaker and K. Walker-Bone, “Shoulder disorders and occupation,” Best practice & research Clinical rheumatology, vol. 29, no. 3, pp. 405-423, 2015.
 
[2]  J. Perry, “Normal upper extremity kinesiology,” Physical therapy, vol. 58, no. 3, pp. 265-278, 1978.
 
[3]  C. Sicuri, G. Porcellini, and G. Merolla, “Robotics in shoulder rehabilitation,” Muscles, ligaments and tendons journal, vol. 4, no. 2, p. 207, 2014.
 
[4]  N. D. Boardman III, R. H. Cofield, K. A. Bengtson, R. Little, M. C. Jones, and C. M. Rowland, “Rehabilitation after total shoulder arthroplasty,” The Journal of arthroplasty, vol. 16, no. 4, pp. 483-486, 2001.
 
[5]  J. J. Brems, “Rehabilitation following total shoulder arthroplasty.” Clinical orthopedics and related research, no. 307, pp. 70-85, 1994.
 
[6]  D. D. Brown and R. J. Friedman, “Postoperative rehabilitation following total shoulder arthroplasty,” Orthopedic Clinics of North America, vol. 29, no. 3, pp. 535-547, 1998.
 
[7]  M. Hughes and C. S. Neer, “Glenohumeral joint replacement and postoperative rehabilitation,” Physical therapy, vol. 55, no. 8, pp. 850-858, 1975.
 
[8]  S. Jackins, “Postoperative shoulder rehabilitation.” Physical medicine and rehabilitation clinics of North America, vol. 15, no. 3, pp. vi-643, 2004.
 
[9]  R. Argent, A. Daly, and B. Caulfield, “Patient involvement with homebased exercise programs: can connected health interventions influence adherence?” JMIR mHealth and uHealth, vol. 6, no. 3, p. e47, 2018.
 
[10]  S. Bassett, “Measuring patient adherence to physiotherapy,” J Nov Physiother, vol. 2, no. 07, pp. 60-66, 2012.
 
[11]  E. M. Sluijs, G. J. Kok, and J. Van der Zee, “Correlates of exercise compliance in physical therapy,” Physical therapy, vol. 73, no. 11, pp. 771-782, 1993.
 
[12]  H. Mousavi Hondori and M. Khademi, “A review on technical and clinical impact of Microsoft Kinect on physical therapy and rehabilitation,” Journal of medical engineering, vol. 2014, 2014.
 
[13]  G. C. Burdea, “Virtual rehabilitation-benefits and challenges,” Methods of information in medicine, vol. 42, no. 05, pp. 519-523, 2003.
 
[14]  C. for Disease Control, Prevention et al., “Brfss survey data and documentation 2017,” 2019.
 
[15]  D. Corbetta, F. Imeri, and R. Gatti, “Rehabilitation that incorporates virtual reality is more effective than standard rehabilitation for improving walking speed, balance and mobility after stroke: a systematic review,” Journal of physiotherapy, vol. 61, no. 3, pp. 117-124, 2015.
 
[16]  A. Elor, M. Teodorescu, and S. Kurniawan, “Project star catcher: A novel immersive virtual reality experience for upper limb rehabilitation,” ACM Transactions on Accessible Computing (TACCESS), vol. 11, no. 4, p. 20, 2018.
 
[17]  A. Elor, M. Powell, E. Mahmoodi, N. Hawthorne, M. Teodorescu, and S. Kurniawan, “On shooting stars: Comparing cave and hmd immersive virtual reality exergaming for adults with mixed ability,” ACM Transactions on Computing for Healthcare.
 
[18]  H. G. Hoffman, W. J. Meyer III, M. Ramirez, L. Roberts, E. J. Seibel, B. Atzori, S. R. Sharar, and D. R. Patterson, “Feasibility of articulated arm mounted oculus rift virtual reality goggles for adjunctive pain control during occupational therapy in pediatric burn patients,” Cyberpsychology, Behavior, and Social Networking, vol. 17, no. 6, pp. 397-401, 2014.
 
[19]  H. G. Hoffman, G. T. Chambers, W. J. Meyer, L. L. Arceneaux, W. J. Russell, E. J. Seibel, T. L. Richards, S. R. Sharar, and D. R. Patterson, “Virtual reality as an adjunctive non-pharmacologic analgesic for acute burn pain during medical procedures,” Annals of Behavioral Medicine, vol. 41, no. 2, pp. 183-191, 2011.
 
[20]  P. J. Standen and D. J. Brown, “Virtual reality in the rehabilitation of people with intellectual disabilities,” Cyberpsychology & behavior, vol. 8, no. 3, pp. 272-282, 2005.
 
[21]  J. Diemer, G. W. Alpers, H. M. Peperkorn, Y. Shiban, and A. Muhlberger, “The impact of perception and presence on emotional¨ reactions: a review of research in virtual reality,” Frontiers in psychology, vol. 6, 2015.
 
[22]  P. J. Costello, Health and safety issues associated with virtual reality: a review of current literature. Advisory Group on Computer Graphics, 1997.
 
[23]  D. E. Levac, S. M. Glegg, H. Sveistrup, H. Colquhoun, P. Miller, H. Finestone, V. DePaul, J. E. Harris, and D. Velikonja, “Promoting therapists use of motor learning strategies within virtual reality-based stroke rehabilitation,” PloS one, vol. 11, no. 12, p. e0168311, 2016.
 
[24]  M. Beccue and C. Wheelock, “Research report: Virtual reality for consumer markets,” Tractica Research, Tech. Rep., Q4 2016. [Online]. Available: https://www.tractica.com/research/virtual-reality-for-consumer-markets/.
 
[25]  A. Elor, S. Kurniawan, and M. Teodorescu, “Towards an immersive virtual reality game for smarter post-stroke rehabilitation,” in 2018 IEEE International Conference on Smart Computing (SMARTCOMP). IEEE, 2018, pp. 219-225.
 
[26]  G. Riva, C. Botella, P. Legeron, and G. Optale, “13 immersive virtual´ telepresence: Virtual reality meets ehealth,” 2004.
 
[27]  P. Lindner, A. Miloff, S. Fagernas, J. Andersen, M. Sigeman, G. An-¨ dersson, T. Furmark, and P. Carlbring, “Therapist-led and self-led onesession virtual reality exposure therapy for public speaking anxiety with consumer hardware and software: A randomized controlled trial,” Journal of anxiety disorders, vol. 61, pp. 45-54, 2019.
 
[28]  A. Won, J. Bailey, J. Bailenson, C. Tataru, I. Yoon, and B. Golianu, “Immersive virtual reality for pediatric pain,” Children, vol. 4, no. 7, p. 52, 2017.
 
[29]  L. Li, F. Yu, D. Shi, J. Shi, Z. Tian, J. Yang, X. Wang, and Q. Jiang, “Application of virtual reality technology in clinical medicine,” American journal of translational research, vol. 9, no. 9, p. 3867, 2017.
 
[30]  K. E. Laver, B. Lange, S. George, J. E. Deutsch, G. Saposnik, and M. Crotty, “Virtual reality for stroke rehabilitation,” Cochrane database of systematic reviews, no. 11, 2017.
 
[31]  S. L. Delp, F. C. Anderson, A. S. Arnold, P. Loan, A. Habib, C. T. John, E. Guendelman, and D. G. Thelen, “Opensim: open-source software to create and analyze dynamic simulations of movement,” IEEE transactions on biomedical engineering, vol. 54, no. 11, pp. 1940-1950, 2007.
 
[32]  A. Rajagopal, C. L. Dembia, M. S. DeMers, D. D. Delp, J. L. Hicks, and S. L. Delp, “Full-body musculoskeletal model for muscle-driven simulation of human gait,” IEEE Transactions on Biomedical Engineering, vol. 63, no. 10, pp. 2068-2079, 2016.
 
[33]  E. M. Arnold, S. R. Hamner, A. Seth, M. Millard, and S. L. Delp, “How muscle fiber lengths and velocities affect muscle force generation as humans walk and run at different speeds,” Journal of Experimental Biology, vol. 216, no. 11, pp. 2150-2160, 2013.
 
[34]  T. K. Uchida, J. L. Hicks, C. L. Dembia, and S. L. Delp, “Stretching your energetic budget: how tendon compliance affects the metabolic cost of running,” PloS one, vol. 11, no. 3, p. e0150378, 2016.
 
[35]  H. Baskar and S. M. R. Nadaradjane, “Minimization of metabolic cost of muscles based on human exoskeleton modeling: a simulation,” Int. J. Biomed. Eng. Sci, vol. 3, no. 4, p. 9, 2016.
 
[36]  B. J. de Kruif, E. Schmidhauser, K. S. Stadler, and L. W. O’Sullivan, “Simulation architecture for modelling interaction between user and elbow-articulated exoskeleton,” Journal of Bionic Engineering, vol. 14, no. 4, pp. 706-715, 2017.
 
[37]  G. S. Virk, U. Haider, I. Nyoman, N. Masud, I. Mamaev, P. Hopfgarten, and B. Hein, “Design of exo-legs exoskeletons,” in ASSISTIVE ROBOTICS: Proceedings of the 18th International Conference on CLAWAR 2015. World Scientific, 2016, pp. 59-66.
 
[38]  D. S. Pina, A. A. Fernandes, R. N. Jorge, and J. Gabriel, “Designing the mechanical frame of an active exoskeleton for gait assistance,” Advances in Mechanical Engineering, vol. 10, no. 2, p. 1687814017743664, 2018.
 
[39]  J. A. Dienes, X. Hu, K. D. Janson, C. Slater, E. A. Dooley, G. J. Christ, and S. D. Russell, “Analysis and modeling of rat gait biomechanical deficits in response to volumetric muscle loss injury,” Frontiers in Bioengineering and Biotechnology, vol. 7, p. 146, 2019.
 
[40]  M. Marieswaran, A. Sikidar, A. Goel, D. Joshi, and D. Kalyanasundaram, “An extended opensim knee model for analysis of strains of connective tissues,” Biomedical engineering online, vol. 17, no. 1, p. 42, 2018.
 
[41]  Z. F. Lerner, B. C. Gadomski, A. K. Ipson, K. K. Haussler, C. M. Puttlitz, and R. C. Browning, “Modulating tibiofemoral contact force in the sheep hind limb via treadmill walking: Predictions from an opensim musculoskeletal model,” Journal of Orthopaedic Research, vol. 33, no. 8, pp. 1128-1133, 2015.
 
[42]  J. W. Rankin, J. Rubenson, and J. R. Hutchinson, “Inferring muscle functional roles of the ostrich pelvic limb during walking and running using computer optimization,” Journal of the Royal Society Interface, vol. 13, no. 118, p. 20160035, 2016.
 
[43]  J. M. Burnfield, K. R. Josephson, C. M. Powers, and L. Z. Rubenstein, “The influence of lower extremity joint torque on gait characteristics in elderly men,” Archives of physical medicine and rehabilitation, vol. 81, no. 9, pp. 1153-1157, 2000.
 
[44]  L. Ballaz, M. Raison, C. Detrembleur, G. Gaudet, and M. Lemay, “Joint torque variability and repeatability during cyclic flexion-extension of the elbow,” BMC sports science, medicine and rehabilitation, vol. 8, no. 1, p. 8, 2016.
 
[45]  A. K. Gillawat and H. J. Nagarsheth, “Human upper limb joint torque minimization using genetic algorithm,” in Recent Advances in Mechanical Engineering. Springer, 2020, pp. 57-70.
 
[46]  K. Kiguchi and Y. Hayashi, “An emg-based control for an upper-limb power-assist exoskeleton robot,” IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), vol. 42, no. 4, pp. 1064-1071, 2012.
 
[47]  D. H. Perrin, R. J. Robertson, and R. L. Ray, “Bilateral isokinetic peak torque, torque acceleration energy, power, and work relationships in athletes and nonathletes,” Journal of Orthopaedic & Sports Physical Therapy, vol. 9, no. 5, pp. 184-189, 1987.
 
[48]  J. Hamill and K. M. Knutzen, Biomechanical basis of human movement. Lippincott Williams & Wilkins, 2006.
 
[49]  M. T. Farrell and H. Herr, “Angular momentum primitives for human turning: Control implications for biped robots,” in Humanoids 2008-8th IEEE-RAS International Conference on Humanoid Robots. IEEE, 2008, pp. 163-167.
 
[50]  S. M. Bruijn, P. Meyns, I. Jonkers, D. Kaat, and J. Duysens, “Control of angular momentum during walking in children with cerebral palsy,” Research in developmental disabilities, vol. 32, no. 6, pp. 2860-2866, 2011.
 
[51]  C. Nott, R. R. Neptune, and S. Kautz, “Relationships between frontalplane angular momentum and clinical balance measures during poststroke hemiparetic walking,” Gait & posture, vol. 39, no. 1, pp. 129-134, 2014.
 
[52]  R. R. Neptune and C. P. McGowan, “Muscle contributions to wholebody sagittal plane angular momentum during walking,” Journal of biomechanics, vol. 44, no. 1, pp. 6-12, 2011.
 
[53]  J. F. Lubar, “Discourse on the development of eeg diagnostics and biofeedback for attention-deficit/hyperactivity disorders,” Biofeedback and Self-regulation, vol. 16, no. 3, pp. 201-225, 1991.
 
[54]  J. Lubar, M. Swartwood, J. Swartwood, and D. Timmermann, “Quantitative eeg and auditory event-related potentials in the evaluation of attention-deficit/hyperactivity disorder: Effects of methylphenidate and implications for neurofeedback training,” Journal of Psychoeducational Assessment, vol. 34, pp. 143-160, 1995.
 
[55]  G. Deuschl, A. Eisen et al., “Recommendations for the practice of clinical neurophysiology: guidelines of the international federation of clinical neurophysiology,” 1999.
 
[56]  M. Lotan, S. Yalon-Chamovitz, and P. L. T. Weiss, “Improving physical fitness of individuals with intellectual and developmental disability through a virtual reality intervention program,” Research in developmental disabilities, vol. 30, no. 2, pp. 229-239, 2009.
 
[57]  H. D. Critchley, “Electrodermal responses: what happens in the brain,” The Neuroscientist, vol. 8, no. 2, pp. 132-142, 2002.
 
[58]  W. Boucsein, Electrodermal activity. Springer Science & Business Media, 2012.
 
[59]  R. Ramirez and Z. Vamvakousis, “Detecting emotion from eeg signals using the emotive epoc device,” in International Conference on Brain Informatics. Springer, 2012, pp. 175-184.
 
[60]  V. N. Salimpoor, M. Benovoy, G. Longo, J. R. Cooperstock, and R. J. Zatorre, “The rewarding aspects of music listening are related to degree of emotional arousal,” PloS one, vol. 4, no. 10, p. e7487, 2009.
 
[61]  M. S. Cameirao, I. B. S. Bermudez, E. Duarte Oller, and P. F. Verschure,´ “The rehabilitation gaming system: a review,” Stud Health Technol Inform, vol. 145, no. 6, 2009.
 
[62]  R. W. Picard, Affective computing. MIT press, 2000.
 
[63]  D. Novak, J. Ziherl, A. Olensek, M. Milavec, J. Podobnik, M. Mihelj, and M. Munih, “Psychophysiological responses to robotic rehabilitation tasks in stroke,” IEEE Transactions on neural systems and rehabilitation engineering, vol. 18, no. 4, pp. 351-361, 2010.
 
[64]  D. Novak, M. Mihelj, J. Ziherl, A. Olensek, and M. Munih, “Psychophysiological measurements in a biocooperative feedback loop for upper extremity rehabilitation,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 19, no. 4, pp. 400-410, 2011.
 
[65]  C. R. Guerrero, J. C. F. Marinero, J. P. Turiel, and V. Munoz, “Using hu-˜ man state aware robots to enhance physical human-robot interaction in a cooperative scenario,” Computer methods and programs in biomedicine, vol. 112, no. 2, pp. 250-259, 2013.
 
[66]  A. Koenig, D. Novak, X. Omlin, M. Pulfer, E. Perreault, L. Zimmerli, M. Mihelj, and R. Riener, “Real-time closed-loop control of cognitive load in neurological patients during robot-assisted gait training,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 19, no. 4, pp. 453-464, 2011.
 
[67]  A. Elor, S. Lessard, M. Teodorescu, and S. Kurniawan, “Project butterfly: Synergizing immersive virtual reality with actuated soft exosuit for upper-extremity rehabilitation,” in 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 2019, pp. 1448-1456.
 
[68]  I. D. O. NaturalPoint, “Optitrack - industry leading precision motion capture and 3d tracking systems for video game design, animation, virtual reality, robotics, and movement sciences.” O. M. C. Systems, Ed., 23.01.2020. [Online]. Available: https://optitrack.com/.
 
[69]  InteraXon, “Featured research with muse,” M. Research, Ed., 29.06.2019. [Online]. Available: https://choosemuse.com/muse-research/ ,developer.choosemuse.com/tools/available-data.
 
[70]  J. W. Cooley and J. W. Tukey, “An algorithm for the machine calculation of complex fourier series,” Mathematics of computation, vol. 19, no. 90, pp. 297-301, 1965.
 
[71]  N. Kovacevic, P. Ritter, W. Tays, S. Moreno, and A. R. McIntosh, “‘my virtual dream’ : Collective neurofeedback in an immersive art environment,” PloS one, vol. 10, no. 7, p. e0130129, 2015.
 
[72]  O. E. Krigolson, C. C. Williams, A. Norton, C. D. Hassall, and F. L. Colino, “Choosing muse: Validation of a low-cost, portable eeg system for erp research,” Frontiers in neuroscience, vol. 11, p. 109, 2017.
 
[73]  S. Bhayee, P. Tomaszewski, D. H. Lee, G. Moffat, L. Pino, S. Moreno, and N. A. Farb, “Attentional and affective consequences of technology supported mindfulness training: a randomised, active control, efficacy trial,” BMC psychology, vol. 4, no. 1, p. 60, 2016.
 
[74]  Neulog, “Gsr logger sensor nul-217,” Neulog, Ed., 29.06.2019. [Online]. Available: https://neulog.com/gsr/.
 
[75]  Polar, “Polar oh1: optical heart rate sensor,” Polar, Ed., 29.06.2019. [Online]. Available: https://www.polar.com/us-en/products/accessories/ oh1-optical-heart-rate-sensor.
 
[76]  H. Moore, MATLAB for Engineers.Pearson, 2017.
 
[77]  Unity Technologies, “Unity real-time development platform — 3d, 2d vr ar,” Internet: https://unity.com/ [Jun. 06, 2019], 2019.
 
[78]  C. Jennett, A. L. Cox, P. Cairns, S. Dhoparee, A. Epps, T. Tijs, and A. Walton, “Measuring and defining the experience of immersion in games,” International journal of human-computer studies, vol. 66, no. 9, pp. 641-661, 2008.
 
[79]  G. Rossum, “Python reference manual,” 1995.