1. Introduction

2. The Audio-Visual Fusion Model Used for Assessing the Skill Levels of Piano Performers

Figure 1 shows the framework of researchers' proposal. It consists of three main components: data pre-processing, feature extraction and fusion, and performance evaluation. 

Figure 1. Framework of audio-visual fusion model for piano skills evaluation.

2.1  Data Pre-Processing

2.2 Feature Extraction and Fusion

Visual branch: Specifically, researchers consider ResNet-3D ^[5] to extract the spatio-temporal features of the performance clips from a video sequence. Compared to conventional 3D CNNs, ResNet-3D can effectively capture the spatio-temporal dynamics of the video modality with higher computational efficiency. For ResNet-3D, researchers stack multiple 3D convolutional layers to model motion features in the temporal dimension and utilize 3D pooling layers and fully connected layers for feature descent and combination. In this way, researchers can extract rich visual features from the video data, including information such as the object's shape and color, to capture finger motion patterns. In addition, researchers can utilize a pre-trained model to improve the model's performance.

Aural branch: Information such as the pitch and rhythm of a piano performance is contained in the audio data, and both raw audio waveforms ^{[6] [7]} and spectrograms ^{[8] [9]} can be used to extract the auditory features. Specifically, researchers convert the raw audio data into the corresponding Mel-spectrogram. Researchers then feed the obtained Mel-spectrogram to the auditory network. Further, ResNet-2D ^[10] outperforms the traditional 2D CNN in terms of computational efficiency and feature extraction. Additionally, it can utilize a pre-trained model to improve performance. Therefore, researchers prefer to use ResNet-2D for feature extraction from the Mel-spectrogram. By stacking 2D convolutional layers, researchers can capture the patterns and variations in the audio data in the frequency and time dimensions. 

Multimodal branch: By utilizing the ResNet-3D and ResNet-2D networks, researchers have obtained visual and aural features. To better capture the semantic association and complementary information between the video and audio modalities, researchers adopt a joint representation approach for the features extracted from the video and audio data. This helps to create a more comprehensive and accurate feature representation. 

Aggregation option: It is often advantageous to perform linear operations on the learned features, which enhances the interpretability and expressiveness of the learned features. Linear operations can also be utilized to reduce the dimensionality of the features, which enhances the efficiency and generalization capabilities of the model. Consequently, researchers propose the utilization of linear averaging as the preferred aggregation scheme. Figure 3 shows the details of the linear averaging.

Figure 3. Feature average option.

2.3 Performance Evaluation

In the visual and aural branches, to reduce the dimensionality of the features to 128, researchers pass them through a linear layer, as shown in Figure 4, and finally input them into the prediction layer. In the multimodal branch, researchers' operations are similar to those of others, except that researchers do not back-propagate from the multimodal branch to a separate modal backbone to avoid cross-modal contamination.

Figure 4. The structure of feature fusion.

3. Conclusions

Table 1 and Figure 5 present the results of the unimodal and audio-visual fusion models on the PISA dataset. The audio-visual fusion model achieved better results than the unimodal model. This indicates that the multimodal method can effectively compensate for the inability of the audio model alone to utilize visual information, leading to more accurate and comprehensive evaluation results. 

Figure 5. Performance (% accuracy) of unimodal evaluation.