Your browser does not fully support modern features. Please upgrade for a smoother experience.
Submitted Successfully!
Thank you for your contribution! You can also upload a video entry or images related to this topic. For video creation, please contact our Academic Video Service.
Version Summary Created by Modification Content Size Created at Operation
1 Eng Editorial Office -- 234 2026-09-23 09:16:03 |
2 format correct Catherine Yang Meta information modification 234 2026-09-24 04:43:18 |

Video Upload Options

We provide professional Academic Video Service to translate complex research into visually appealing presentations. Would you like to try it?
Cite
If you have any further questions, please contact Encyclopedia Editorial Office.
Zheng, L. Multimodal Action Recognition. Encyclopedia. Available online: https://encyclopedia.pub/entry/60406 (accessed on 23 October 2026).
Zheng L. Multimodal Action Recognition. Encyclopedia. Available at: https://encyclopedia.pub/entry/60406. Accessed October 23, 2026.
Zheng, Lionel. "Multimodal Action Recognition" Encyclopedia, https://encyclopedia.pub/entry/60406 (accessed October 23, 2026).
Zheng, L. (2026, September 23). Multimodal Action Recognition. In Encyclopedia. https://encyclopedia.pub/entry/60406
Zheng, Lionel. "Multimodal Action Recognition." Encyclopedia. Web. 23 September, 2026.

Multimodal Action Recognition

Edit

Multimodal action recognition is the computational task of identifying and classifying human physical actions by jointly interpreting information streams derived from multiple sensory modalities, typically including RGB video frames, depth maps, skeletal joint coordinates, inertial measurement unit signals, audio, and text descriptions. The core technical premise is that no single sensory channel captures the full discriminative information of a human action, and fusing complementary modalities yields action representations that are more robust to viewpoint variation, occlusion, and illumination change than unimodal approaches [1]. The recognition pipeline consists of modality-specific feature extraction—convolutional neural networks for visual frames, temporal convolutional networks or recurrent models for skeleton sequences, and spectral or temporal encoders for inertial signals—followed by cross-modal fusion that aligns features temporally and semantically before a classifier infers the action label. The fusion architecture may be early-level, combining raw signals before feature extraction, or late-level, combining modality-specific predictions, with intermediate-level feature fusion representing the dominant paradigm in deep-learning-based systems . The conceptual foundation is that human actions are inherently spatiotemporal events characterized by joint motion patterns, body shape deformation, environmental context, and vocal or textual cues, and each modality captures a different projection of this underlying event, such that multimodal fusion approximates the full action manifold more completely than any single projection [2].

multimodal action recognition sensor fusion deep learning human activity recognition
Help expand this entry Share your expertise by adding detailed content and references to improve this entry.
Add content

Human Pose and Action Recognition •  Computer Vision and Pattern Recognition •  Computer Science •  Physical Sciences

References

  1. Sun Z.; Ke Q.; Rahmani H.; Bennamoun M.; Wang G.; Liu J. Human action recognition from various data modalities: A review.. IEEE Trans. Pattern Anal. Mach. Intell. 2023, 45, 3200-3225. [CrossRef]
  2. Yoshua Bengio; Aaron Courville; Pascal Vincent; Representation Learning: A Review and New Perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 2013, 35, 1798-1828. [CrossRef]
More
Upload a video for this entry
Information
Contributor MDPI registered users' name will be linked to their SciProfiles pages. To register with us, please refer to https://encyclopedia.pub/register : Eng Editorial Office
View Times: 5
Entry Collection: Eng
Revisions: 2 times (View History)
Update Date: 24 Sep 2026
Notice
You are not a member of the advisory board for this topic. If you want to update advisory board member profile, please contact office@encyclopedia.pub.
OK
Confirm
Only members of the Encyclopedia advisory board for this topic are allowed to note entries. Would you like to become an advisory board member of the Encyclopedia?
Yes
No
${ textCharacter }/${ maxCharacter }
Submit
Cancel
There is no comment~
${ textCharacter }/${ maxCharacter }
Submit
Cancel
${ selectedItem.replyTextCharacter }/${ selectedItem.replyMaxCharacter }
Submit
Cancel
Confirm
Are you sure to Delete?
Yes No
Write an Entry