Work Package 2: Multimodal Perception and Modeling Sample Clauses

Work Package 2: Multimodal Perception and Modeling. 2.1 Overall Summary for Work Package 2: The objective of WP2 is to provide a foundation for multimodal perception and modeling needed for natural interaction and collaboration between humans and intelligent systems. Humans interact with the world using five major senses: sight, hearing, touch, smell, and taste. Almost all interaction with the environment is naturally multimodal, as audio, tactile or paralinguistic cues provide confirmation for physical actions and spoken language interaction. To interact and collaborate with people, intelligent systems must be able to perceive and model humans, human actions, and behaviors, human attention and awareness, human emotions, human language and human social interaction, as well as real-world human environments. Multimodal interaction seeks to fully exploit these parallel channels for perception and action to provide robust, natural interaction and collaboration with intelligent systems. WP2 builds on recent advances in multimodal perception and modeling. Specifically, we define research challenges for natural multimodal interaction, document and augment the existing tools and data sets needed to meet such challenges and demonstrate the use of recent advances in machine learning for building systems. Work on multimodal perception and modeling is organized around the following activities: Learning of multimodal models grounded in physical reality (T2.1), Multimodal perception and modeling of actions, activities and tasks (T2.2), Multimodal perception of awareness, emotions, and attitudes (T2.3), Perception of Social Signals and Social Dynamics (T2.4), Distributed Collaborative Perception and Modeling (T2.5), Dealing with lack of labeled training data (T2.6), and Assembling benchmark datasets (T2.7).