Introduction
This project involves a joint collaboration between the Computer Science Department at California Polytechnic State University in San Luis Obispo and the Intel Corporation. Our project's collaborator from Intel Corp. provided the primary focus and objectives for this project which involves the use of basic web cameras to detect the gesture of throwing virtual objects on a computer. This focus requires the use of computer vision software in attempts to detect the generalized gestures of throwing.
Project Overview
This project is a continuation form the Human Computer Interface course offered the previous quarter and carried into the Knowledge and Usability course offered in Spring Quarter 2014. At which time, the project vision and scope were identified as utilizing a basic web camera in conjunction with the Java implementation of OpenCV, Open source Computer Vision library, to detect the essential hand gestures required for detecting a user’s throwing action. This action of throwing consists of the more general capability to accurately detect the gestures of open and closed hands. Once these two primary gestures can be accurately detected and traced, the application will map them to the mouse input device.
The gesture interface application will continue as the primary demonstration and testing platform of any future detection methods. Some reworking is required to allow for these other methods of gesture detection rather than only utilizing classifiers.
Based on the results from the previous quarters work, using Haar-like cascading classifiers proved unreliable for detecting open and closed hands. This contributed to the redirection of move active analysis and coding on our application side to test using skin tone detection, contour subtraction, and convexity defect detection, with the possibility of combining all of these techniques with classifiers for open and closed hands.
Main Features
For the end user of this application concept, this project entails the detection of their hand gestures and mapping them to interact with their computer system. These primary features include:
- Hand Gesture Detection
- The ability to accurately detect an open or closed hand from the user’s web camera.
- Drag-and-Drop
- The drag-and-drop functionality based on the hand gesture detection (Feature 1) which also enables mapping of the hand gestures to the mouse input device.
- Throw Gesture Detection
- Implementation of the throw gesture relies on trajectory data computed from the drag and drop functionality (Feature 2). Thus, if an object on screen was dragged and let go with a velocity over some defined threshold, the object would then be thrown.
Note: At this time, the specifics of what the thrown object would do are not defined.
- Gesture Interface Application
- The system tray application allows the user to utilize their own hands as the input device to interact with their computers. This application runs as a background system tray application on most platforms and provides the ability to turn on or off the mouse overrides. This software provides both the testing environment and the demonstration application of the other features.