A Stereo Vision-Based Method for Reconstructing 3-D Hand Motion in Real-Time
Corressponding author's email:
ntnhu@hcmiu.edu.vnDOI:
https://doi.org/10.54644/jte.2025.1735Keywords:
Hand Motion Paralysis, Hand Motion Tracking, Stereo Vision-based Hand Motion Analysis, Hand Motion Diagnosis, Clinical Decision Support System for Hand ParalysisAbstract
Hand motion paralysis negatively affects the lives of the involved patients. To recover the hand motions into their normal condition, these patients need to be taken into complex and long-term rehabilitation treatments. During rehabilitation, hand motions with full finger features need to be tracked accurately in 3-D dimension in real-time for analyzing and diagnosing hand motion paralysis. However, most studies tried to track hand motions based on contact sensors. These methods are not user-friendly. Even using contactless sensors, most of them could only detect the hand motions in 2-D image spaces. Consequently, in this study, we developed a stereo vision-based method for detecting and tracking 3-D hand features in real-time. In particular, we employed a convolutional deep neural network (C-DNN) for tracking by detecting hand-finger features. The features were tracked on left and right images captured by a stereo camera system before being reconstructed into 3-D spaces. A meta-validation procedure was conducted to compute the accuracy of the method with various light conditions, skin colors, and hand shapes. As a result, we could successfully track hand motions in real-time with acceptable accuracy. In perspective, we will apply the method for analyzing and diagnosing hand paralysis inside a clinical decision-support system.
Downloads: 0
References
S. L. Gorniak, E. D. Collins, K. Goldie Staines, F. A. Brooks, and R. V Young, “The Impact of Musical Training on Hand Biomechanics in String Musicians.,” Hand (N. Y)., vol. 14, no. 6, pp. 823–829, Nov. 2019. DOI: https://doi.org/10.1177/1558944718772388
S. Brorsson, A. Nilsdotter, E. Pedersen, A. Bremander, and C. Thorstensson, “Relationship between finger flexion and extension force in healthy women and women with rheumatoid arthritis,” J. Rehabil. Med., vol. 44, no. 7, pp. 605–608, 2012. DOI: https://doi.org/10.2340/16501977-0986
A. Borboni et al., “Robot-Assisted Rehabilitation of Hand Paralysis After Stroke Reduces Wrist Edema and Pain: A Prospective Clinical Trial,” J. Manipulative Physiol. Ther., vol. 40, no. 1, pp. 21–30, Jan. 2017. DOI: https://doi.org/10.1016/j.jmpt.2016.10.003
V. Nazari, M. Pouladian, Y. P. Zheng, and M. Alam, “A Compact and Lightweight Rehabilitative Exoskeleton to Restore Grasping Functions for People with Hand Paralysis,” Sensors, vol. 21, no. 20, p. 6900, Oct. 2021. DOI: https://doi.org/10.3390/s21206900
A. Schiavio and M. Benedek, “Dimensions of Musical Creativity,” Front. Neurosci., vol. 14, Nov. 2020. DOI: https://doi.org/10.3389/fnins.2020.578932
A. Floel and L. G. Cohen, “Translational Studies in Neurorehabilitation: From Bench to Bedside,” Cogn. Behav. Neurol., vol. 19, no. 1, pp. 1–10, Mar. 2006. DOI: https://doi.org/10.1097/00146965-200603000-00001
L. Wade, L. Needham, P. McGuigan, and J. Bilzon, “Applications and limitations of current markerless motion capture methods for clinical gait biomechanics,” PeerJ, vol. 10, p. e12995, Feb. 2022. DOI: https://doi.org/10.7717/peerj.12995
C. Mummadi et al., “Real-Time and Embedded Detection of Hand Gestures with an IMU-Based Glove,” Informatics, vol. 5, no. 2, p. 28, Jun. 2018. DOI: https://doi.org/10.3390/informatics5020028
D. Zhang et al., “Fine-Grained and Real-Time Gesture Recognition by Using IMU Sensors,” IEEE Trans. Mob. Comput., vol. 22, no. 4, pp. 2177–2189, Apr. 2023. DOI: https://doi.org/10.1109/TMC.2021.3120475
K. Kwon, H. Zhang, and F. Dornaika, “Hand pose recovery with a single video camera,” in Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164), vol. 2, pp. 1194–1200. DOI: https://doi.org/10.1109/ROBOT.2001.932773
Muslikhin, J. R. Horng, S. Y. Yang, and M. S. Wang, “Object Localization and Depth Estimation for Eye-in-Hand Manipulator Using Mono Camera,” IEEE Access, vol. 8, pp. 121765–121779, 2020. DOI: https://doi.org/10.1109/ACCESS.2020.3006843
C. S. Lee, S. Chun, and S. W. Park, “Tracking hand rotation and various grasping gestures from an IR camera using extended cylindrical manifold embedding,” Comput. Vis. Image Underst., vol. 117, no. 12, pp. 1711–1723, Dec. 2013. DOI: https://doi.org/10.1016/j.cviu.2013.08.006
J. Zhang, J. Jiao, M. Chen, L. Qu, X. Xu, and Q. Yang, “3D Hand Pose Tracking and Estimation Using Stereo Matching,” Oct. 2016.
S. Jiang et al., “In-Hand 3D Object Reconstruction from a Monocular RGB Video,” Dec. 2023. DOI: https://doi.org/10.1609/aaai.v38i3.28029
T. N. Nguyen, A. Ballit, and T. T. Dao, “A Novel Stereo Camera Fusion Scheme for Generating and Tracking Real-Time 3-D Patient-Specific Head/Face Kinematics and Facial Muscle Movements,” IEEE Sens. J., vol. 23, no. 9, pp. 9889–9897, May 2023. DOI: https://doi.org/10.1109/JSEN.2023.3259473
R. M. Jans, A. S. Green, and L. J. Koerner, “Characterization of a Miniaturized IR Depth Sensor With a Programmable Region-of-Interest That Enables Hazard Mapping Applications,” IEEE Sens. J., vol. 20, no. 10, pp. 5213–5220, May 2020. DOI: https://doi.org/10.1109/JSEN.2020.2971595
F. Alqahtani, J. Banks, V. Chandran, and J. Zhang, “3D Face Tracking Using Stereo Cameras: A Review,” IEEE Access, vol. 8, pp. 94373–94393, 2020. DOI: https://doi.org/10.1109/ACCESS.2020.2994283
Y. F. Ji, “Vision-based sensing for assessing and monitoring civil infrastructures,” in Sensor Technologies for Civil Infrastructures, Elsevier, 2022, pp. 309–333. DOI: https://doi.org/10.1016/B978-0-08-102696-0.00016-6
A. Khan, M. M. Hossain, A. Covaci, K. Sirlantzis, and C. Xu, “Light field imaging technology for virtual reality content creation: A review,” IET Image Process., vol. 18, no. 11, pp. 2817–2837, Sep. 2024. DOI: https://doi.org/10.1049/ipr2.13144
E. Adil, M. Mikou, and A. Mouhsen, “A novel algorithm for distance measurement using stereo camera,” CAAI Trans. Intell. Technol., vol. 7, no. 2, pp. 177–186, Jun. 2022. DOI: https://doi.org/10.1049/cit2.12098
F. Zhang et al., “MediaPipe Hands: On-device Real-time Hand Tracking,” Jun. 2020.
A. Zelinsky, “Learning OpenCV---Computer Vision with the OpenCV Library (Bradski, G.R. et al.; 2008)[On the Shelf],” IEEE Robot. Autom. Mag., vol. 16, no. 3, pp. 100–100, Sep. 2009. DOI: https://doi.org/10.1109/MRA.2009.933612
J. Dongarra et al., “The Singular Value Decomposition: Anatomy of Optimizing an Algorithm for Extreme Scale,” SIAM Rev., vol. 60, no. 4, pp. 808–865, Jan. 2018. DOI: https://doi.org/10.1137/17M1117732
J. Romero, D. Tzionas, and M. J. Black, “Embodied hands,” ACM Trans. Graph., vol. 36, no. 6, pp. 1–17, Dec. 2017. DOI: https://doi.org/10.1145/3130800.3130883
S. P. Kildare and K. Malone, “Skeletal Anatomy of the Hand,” Hand Clin., vol. 29, no. 4, pp. 459–471, Nov. 2013. DOI: https://doi.org/10.1016/j.hcl.2013.08.001
C. A. Moran, “Anatomy of the Hand,” Phys. Ther., vol. 69, no. 12, pp. 1007–1013, Dec. 1989. DOI: https://doi.org/10.1093/ptj/69.12.1007
H. Chim, “Hand and Wrist Anatomy and Biomechanics: A Comprehensive Guide,” Plast. Reconstr. Surg., vol. 140, no. 4, pp. 865–865, Oct. 2017. DOI: https://doi.org/10.1097/PRS.0000000000003745
K. Pulli, A. Baksheev, K. Kornyakov, and V. Eruhimov, “Real-time computer vision with OpenCV,” Commun. ACM, vol. 55, no. 6, pp. 61–69, Jun. 2012. DOI: https://doi.org/10.1145/2184319.2184337
L. C. Shum, B. A. Valdés, and H. M. Van der Loos, “Determining the Accuracy of Oculus Touch Controllers for Motor Rehabilitation Applications Using Quantifiable Upper Limb Kinematics: Validation Study,” JMIR Biomed. Eng., vol. 4, no. 1, p. e12291, Jun. 2019. DOI: https://doi.org/10.2196/12291
H. A. Mercado, A. P. Vanegas, and A. G. Marrugo, “Robust 3D surface recovery by applying a focus criterion in white light scanning interference microscopy,” Appl. Opt., vol. 58, no. 5, p. A101, Feb. 2019. DOI: https://doi.org/10.1364/AO.58.00A101
Y. J. Park and C. Y. Yi, “Time of Flight Distance Sensor–Based Construction Equipment Activity Detection Method,” Appl. Sci., vol. 14, no. 7, p. 2859, Mar. 2024. DOI: https://doi.org/10.3390/app14072859
A. Karim and J. Y. Andersson, “Infrared detectors: Advances, challenges and new technologies,” IOP Conf. Ser. Mater. Sci. Eng., vol. 51, p. 012001, Dec. 2013. DOI: https://doi.org/10.1088/1757-899X/51/1/012001
K. Saleh, S. Szenasi, and Z. Vamossy, “Occlusion Handling in Generic Object Detection: A Review,” in 2021 IEEE 19th World Symposium on Applied Machine Intelligence and Informatics (SAMI), 2021, pp. 000477–000484. DOI: https://doi.org/10.1109/SAMI50585.2021.9378657
S. Dill et al., “Accuracy Evaluation of 3D Pose Reconstruction Algorithms Through Stereo Camera Information Fusion for Physical Exercises with MediaPipe Pose,” Sensors, vol. 24, no. 23, p. 7772, Dec. 2024. DOI: https://doi.org/10.3390/s24237772
Downloads
Published
How to Cite
Issue
Section
Categories
License
Copyright (c) 2025 Journal of Technical Education Science
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright © JTE.
