Reference Number: DEVCOM-098
Project Description
The Army seeks to discover novel mechanisms that provide context and insight into how individuals and teams understand complex real-world environments, enabling commanders and intelligent systems to communicate and adapt effectively, thereby improving joint human-system performance. Humans are unique in their ability to communicate, intentionally or unintentionally, with one another through physical and verbal cues. This research area leverages foundational principles from the human and computational sciences to create novel methodologies for effective and efficient real-time multimodal communication between Soldiers and intelligent systems. Opportunistic sensing is one means to address this challenge. Opportunistic sensing refers to the process of obtaining operational data required to infer mission, train algorithms, and quantify unit effectiveness through passive sensing from tasks and behaviors that Soldiers naturally perform without adding burden or hindering performance. Algorithms and models based on human data collected via opportunistic sensing can be used to inform more effective communications between Humans and intelligent system development and command.
In the STRIKE study, we leverage opportunistic sensing, utilizing eye-tracking and inertial measurement units (IMUs), to begin constructing models of Soldier engagement and response during an immersive virtual reality visual search task. These models will classify elements of the tactical readiness cycle, specifically when a Soldier detects, mitigates, and confirms a threat (see the following paragraph for a breakdown of the stages). We will then determine a robust reduced sensor set that will be more applicable in field environments. A robust reduced sensor set and model can be utilized to test Soldier engagement and readiness in future scenarios such as endeavors in less restrictive environments (real-life maneuver scenarios in the field), in multi-human teaming scenarios (where too many sensors would be overburdensome), or integration in human-autonomy teams.
To achieve this effort, we first need to create a ground-truth model of our video, capturing all our movement data. An ideal solution would be a customized graphical user interface (GUI) application that allows for manual labeling of STRIKE video data for ground-truthing purposes. Such a solution should include the addition of computer vision to detect distinct poses in the video files initially. Human raters should then be able to provide feedback and manually label tactical response (military) poses within the video (e.g., a Soldier standing, a Soldier with a weapon at low ready or ready-up, etc.). Human input should then be integrated to retrain the video pose detection model for refinement, enabling it to identify human military poses in subsequent videos.
Expected Outcome(s)
A customized app with computer vision and manual labeling GUI for ground-truthing STRIKE videos that will aid in our ability to validate our STRIKE reduced sensor set model. Designers can use https://learnopencv.com/tag/open-pose/ (or a similar resource) as a starting point. The value added is the subject matter expertise to refine pose classification into military-relevant components. The end goal would be to create an OpenPoseArmy model on data labeled from STRIKE.
Technical Skills
- Computer Vision algorithm development
- Pose estimation models
- Video processing
- Software Development
- Proficiency in programming languages (like Python or C++)
- Proficiency in GUI Frameworks”
