Camera-based Gesture Control of a Robotic Arm using Real-time Hand Tracking
Keywords:
Computer vision, Embedded systems, Gesture recognition, Human–machine interaction, Real-time systems, Robotic arm control, Vision-based controlAbstract
Human–robot interaction has become an essential area of research in industrial automation, healthcare, assistive technology, and smart manufacturing. This paper presents a camera-based gesture control system for a robotic arm using real-time hand tracking, providing an intuitive and contactless alternative to conventional control interfaces such as joysticks, gloves, and remote controllers. The proposed system utilizes a standard RGB camera to capture live video of hand movements, while computer vision techniques and real-time hand landmark detection are employed to recognize predefined gestures accurately. The detected hand gestures are processed using OpenCV and MediaPipe-based algorithms, which extract finger positions and hand orientation to generate corresponding motion commands for the robotic arm. These commands are transmitted to a microcontroller that actuates servo motors, enabling precise movement of the robotic arm and gripper operations such as opening, closing, lifting, and object manipulation. The markerless approach eliminates the need for wearable sensors or specialized hardware, thereby reducing system cost, improving user comfort, and simplifying deployment. Experimental evaluation demonstrates reliable gesture recognition with low processing latency, allowing smooth and responsive robotic arm operation under normal lighting conditions. The proposed system exhibits high accuracy, ease of implementation, scalability, and adaptability to various robotic platforms. It offers significant potential for applications in industrial automation, remote object handling, healthcare assistance, rehabilitation, education, hazardous environment operations, and smart human–machine interaction. Future enhancements may include deep learning-based dynamic gesture recognition, three-dimensional hand pose estimation, integration with Internet of Things (IoT) platforms, cloud robotics, and augmented reality interfaces to further improve system intelligence, robustness, and operational flexibility.
References
X. Wang, H. Shen, H. Yu, J. Guo, and X. Wei, “Hand and arm gesture-based human–robot interaction: A review,” In Proceedings of the 6th International Conference on Algorithms, Computing and Systems, pp. 1–7, 2022.
M. A. Goodrich and A. C. Schultz, “Human–robot interaction: A survey,” Foundations and Trends® in Human–Computer Interaction, vol. 1, no. 3, pp. 203–275, 2008.
F. Zhang, V. Bazarevsky, A. Vakunov, A. Tkachenka, G. Sung, C.L. Chang, and M. Grundmann, “Mediapipe hands: On-device real-time hand tracking,” arXiv:2006.10214, 2020.
P. Molchanov, S. Gupta, K. Kim, and J. Kautz, “Hand gesture recognition with 3D convolutional neural networks,” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1–7, 2015.
Z. Yu, C. Lu, Y. Zhang, and L. Jing, “Gesture-controlled robotic arm for agricultural harvesting using a data glove with bending sensor and OptiTrack systems,” Micromachines, vol. 15, no. 7, p. 918, 2024.
R. Barua and S. Datta, “Modernization of robotics application in the 21st century: A review,” Journal of Mechanical Robotics, vol. 5, pp. 26–36, 2020.
V. Alcácer and V. Cruz-Machado, “Scanning the industry 4.0: A literature review on technologies for manufacturing systems,” Engineering Science and Technology, an International Journal, vol. 22, no. 3, pp. 899–919, 2019.
M. Hermann, T. Pentek, and B. Otto, “Design principles for Industrie 4.0 scenarios,” In 2016 49th Hawaii International Conference on System Sciences (HICSS), pp. 3928-3937, 2016.
M. Humayun, “Industrial Revolution 5.0 and the role of cutting edge technologies,” International Journal of Advanced Computer Science and Applications, vol. 12, no. 12, 2021.
Z. Lei, J. Shi, Z. Luo, M. Cheng, and J. Wan, “Intelligent manufacturing from the perspective of Industry 5.0: Application review and prospects,” IEEE Access, vol. 12, pp. 167436–167451, 2024.
Y. Li, J. Lou, Z. Cai, P. Zheng, H. Wu, and X. Wang, “An interactive gesture control system for collaborative manipulator based on Leap Motion Controller,” Advances in Mechanical Engineering, vol. 16, no. 5, p. 16878132241253101, 2024.
G. Angelidis and L. Bampis, “Gesture-controlled robotic arm for small assembly lines,” Machines, vol. 13, no. 3, p. 182, 2025.
W. Shi, J. Cao, Q. Zhang, Y. Li, and L. Xu, “Edge computing: Vision and challenges,” IEEE Internet of Things Journal, vol. 3, no. 5, pp. 637–646, 2016.
Y. Taigman, M. Yang, M. Ranzato, and L. Wolf, “Deepface: Closing the gap to human-level performance in face verification,” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1701-1708, 2014.
P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, vol.1, pp. I-I, 2001.
Gautam, A. Ferreira, and L. Dunai, “Revolutionizing prosthetic hand control using noninvasive sensors and intelligent algorithms: A comprehensive review,” Computers & Electrical Engineering, vol. 102, p. 108200, 2025.
