Multimodal action recognition is the computational task of identifying and classifying human physical actions by jointly interpreting information streams derived from multiple sensory modalities, typically including RGB video frames, depth maps, skeletal joint coordinates, inertial measurement unit signals, audio, and text descriptions. The core technical premise is that no single sensory channel captures the full discriminative information of a human action, and fusing complementary modalities yields action representations that are more robust to viewpoint variation, occlusion, and illumination change than unimodal approaches [1]. The recognition pipeline consists of modality-specific feature extraction—convolutional neural networks for visual frames, temporal convolutional networks or recurrent models for skeleton sequences, and spectral or temporal encoders for inertial signals—followed by cross-modal fusion that aligns features temporally and semantically before a classifier infers the action label. The fusion architecture may be early-level, combining raw signals before feature extraction, or late-level, combining modality-specific predictions, with intermediate-level feature fusion representing the dominant paradigm in deep-learning-based systems [2]. The conceptual foundation is that human actions are inherently spatiotemporal events characterized by joint motion patterns, body shape deformation, environmental context, and vocal or textual cues, and each modality captures a different projection of this underlying event, such that multimodal fusion approximates the full action manifold more completely than any single projection [23].
Human Pose and Action Recognition • Computer Vision and Pattern Recognition • Computer Science • Physical Sciences