Encyclopedia
Please note this is a comparison between Version 1 by Banoth Thulasya Naik and Version 2 by Conner Chen.
Automatic analysis of video in sports is a possible solution to the demands of fans and professionals for various kinds of information. Analyzing videos in sports has provided a wide range of applications, which include player positions, extraction of the ball’s trajectory, content extraction, and indexing, summarization, detection of highlights, on-demand 3D reconstruction, animations, generation of virtual view, editorial content creation, virtual content insertion, visualization and enhancement of content, gameplay analysis and evaluations, identifying player’s actions, referee decisions and other fundamental elements required for the analysis of a game.
Recent developments in video analysis of sports have a focus on the features of computer vision techniques, which are used to perform certain operations for which these are assigned, such as detailed complex analysis such as detection and classification of each player based on their team in every frame or by recognizing the jersey number to classify players based on their team will help to classify various events where the player is involved. In higher-level analysis, such as tracking the player or ball, many more such evaluations are to be considered for the evaluation of a player’s skills, detecting the team’s strategies, events and the formation of tactical positions such as midfield analysis in various sports such as soccer, basketball, and also various sports vision applications such as smart assistants, virtual umpires, assistance coaches. A higher-level semantic interpretation is an effective substitute, especially in situations when reduced human intervention and real-time analysis are desired for the exploitation of the delivered system outputs.
- sports
- ball detection
- player tracking
- artificial intelligence in sports
- computer vision
- embedded paltforms
1. Basketball
Basketball is a sport played between two teams consisting of five players each. The task of this sport is to score more points than the opponent. This sport has several activities with the ball such as passing, throwing, bouncing, batting, or rolling the ball from one player to another. Physical contact with an opponent player may be a foul if the contact impedes the players’ desired movement. The advancements in computer vision techniques have effectively employed fully automated systems to replace the manual analysis of basketball sports. Recognizing the player’s action and classifying the events [1][2][3][29,30,31] in basketball videos helps to analyze the player’s performance. Player/ball detection and tracking in basketball videos are carried out in [4][5][6][7][8][9][32,33,34,35,36,37] but fail in assigning specific identification to avoid identity switching among the players when they cross. By estimating the pose of the player, the trajectory of the ball [10][11][38,39] is estimated from various distances to the basket. By recognizing and classifying the referee’s signals [12][40], player behavior can be assessed and highlights of the game can be extracted [13][41]. The behavior of a basketball team [14][42] can be characterized by the dynamics of space creation presented in [15][16][17][18][19][20][43,44,45,46,47,48] that works to counteract space creation dynamics with a defensive play presented in [21][49]. By detecting the specific location of the player and ball in the basketball court, the player movement can be predicted [22][50] and the ball trajectory [23][24][25][51,52,53] can be generated in three dimensions which is a complicated task. It is also necessary to study the extraction of basketball players’ shooting motion trajectory, combined with the image feature analysis method of basketball shooting, to reconstruct and quantitatively track the basketball players’ shooting motion trajectory [26][27][28][29][54,55,56,57]. However, it is difficult to analyze the game data for each play such as the ball tracking or motion of the players in the game, because the situation of the game changes rapidly, and the structure of the data is complicated. Therefore, it is necessary to analyze the real-time gameplay [30][58]. Table 12 summarizes various proposed methodologies used to complete various challenging tasks in basketball sport including their limitations.
Table 12.
Studies in basketball.
| Studies in Basketball | ||||
|---|---|---|---|---|
| Ref. | Problem Statement | Proposed Methodology | Precision and Performance Characteristics | Limitations and Remarks |
| [3][31] | Recognizing actions of basketball players by using image recognition techniques | Bi-LSTM Sequence2Sequence | The metrics used to evaluate the method are the Spearman rank-order correlation coefficient, Kendall rank-order correlation coefficient, Pearson linear correlation coefficient, and Root Mean Squared Error and achieved 0.921, 0.803, 0.932, and 1.03, respectively. | The methodology failed to recognize difficult actions due to which accuracy is reduced. The accuracy of action recognition can be improved with a deep convolutional neural network. |
| [26][54] | Multi-future trajectory prediction in basketball. | Conditional Variational Recurrent Neural Networks (RNN)—TrajNet++ | The proposed methodology was tested on Average Displacement Error and Final Displacement Error metrics. The methodology is robust if the number it achieves is smaller than 7.01 and 10.61. | The proposed methodology fails to predict the trajectories in the case of uncertain and complex scenarios. As the behavior of the basketball or players is dynamic, belief maps cannot steer future positions. Training the model with a dataset of different events can rectify the failures of predictions. |
| [30][58] | Predicting line-up performance of basketball players by analyzing the situation of the field. | RNN + NN | At the point guard (pg) position 4 candidates were detected and at the center (c) position 3 candidates were detected. The total score of pg candidates is 13.67, 12.96, 13.42, 10.39, and where the total score of c candidates is 10.21, 14.08, and 13.48, respectively. | - |
| [4][32] | Multiplayer tracking in basketball videos | YOLOv3 + Deep-SORT, Faster-RCNN + Deep-SORT, YOLOv3 + DeepMOT, Faster-RCNN + DeepMOT, JDE | Faster-RCNN provides better accuracy than YOLOv3 among baseline detectors. The joint Detection and Embedding method performs better in the accuracy of tracking and computing speed among multi-object tracking methods. | Tracking in specific areas such as severe occlusions and improving detection precision improves the accuracy and computation speed. By adopting frame extraction methods, in terms of speed and accuracy, it can achieve comprehensive performance, which may be an alternative solution. |
| [12][40] | Recognizing the referee signals from real-time videos in a basketball game. | HOG + SVM, LBP + SVM | Achieved an accuracy of 95.6% for referee signal recognition using local binary pattern features and SVM classification. | In the case of a noisy environment, a significant chance of occlusion, an unusual viewing angle, and/or variability of gestures, the performance of the proposed method is not consistent. Detecting jersey color and eliminating all other detected elements in the frame can be the other solution to improve the accuracy of referee signal recognition. |
| [2][30] | Event recognition in basketball videos | CNN | mAP for group activity recognition is 72.1% | The proposed model can recognize the global movement in the video. By recognizing the local movements, the accuracy can be improved. |
| [31][59] | Analyzing the behavior of the player. | CNN + RNN | Achieved an accuracy of 76.5% for four types of actions in basketball videos. | The proposed model gives less accuracy for actions such as passing and fouling. This also gives less accuracy of recognition and prediction on the test dataset compared to the validation dataset. |
| [5][33] | Tracking ball movements and classification of players in a basketball game | YOLO + Joy2019 | Jersey number recognition in terms of Precision achieved is 74.3%. Player recognition in terms of Recall achieved 89.8%. | YOLO confuses the overlapped image for a single player. In the subsequent frame, the tracking ID of the overlapped player is exchanged, which causes wrong player information to be associated with the identified box. |
| [1][29] | Event classifications in basketball videos | CNN + LSTM | The average accuracy using a two-stage event classification scheme achieved 60.96%. | Performance can be improved by introducing information such as individual player pose detection and player location detection |
| [21][49] | Classification of different defensive strategies of basketball payers, particularly when they deviate from their initial defensive action. | KNN, Decision Trees, and SVM | Achieved 69% classification accuracy for automatic defensive strategy identification. | Considered only two defensive strategies `switch’ and `trap’ involved in Basketball. In addition, the alternative method of labeling large Spatio-temporal datasets will also lead to better results. Future research may also consider other defensive strategies such as pick-and-roll and pick-and-pop. |
| [10][38] | Basketball trajectory prediction based on real data and generating new trajectory samples. | BLSTM + MDN | The proposed method performed well in terms of convergence rate and final AUC (91%) and proved deep learning models perform better than conventional models (e.g., GLM, GBM). | To improve the accuracy time series the prediction has to consider. By considering factors such as player cooperation and defense when predicting NBA player positions, the performance of the model can be improved. |
| [23][51] | Generating basketball trajectories. | GRU-CNN | Validated on a hierarchical policy network (HPN) with ground truth and 3 baselines. | The proposed model failed in the trajectory of a three-dimensional basketball match. |
| [13][41] | Score detection, highlights video generation in basketball videos. | BEI+CNN | Automatically analyses the basketball match, detects scoring, and generates highlights. Achieved an accuracy, precision, recall, and F1-score of 94.59%, 96.55%, 92.31%, and 94.38%. | The proposed method is lacking in computation speed which achieved 5 frames per second. Therefore, it cannot be implemented in a real-time basketball match. |
| [6][34] | Multi-person event recognition in basketball videos. | BLSTM | Event classification and event detection were achieved in terms of mean average precision, i.e., 51.6% and 43.5%. | A high-resolution dataset can improve the performance of the model. |
| [16][44] | Player behavior analysis. | RNN | Achieved an accuracy of 80% over offensive strategies. | The methodology fails in many factors such as complexity of interaction, distinctiveness, and diversity of the target classes and other extrinsic factors such as reactions to defense, unexpected events such as fouls, and consistency of executions. |
| [11][39] | Prediction of the 3-point shot in the basketball game | RNN | Evaluated in terms of AUC and achieved 84.30%. | The proposed method fails in the case of high ball velocity and the noisy nature of motion data. |
2. Soccer
Soccer is played using football, and eleven players in two teams compete to deliver the ball into the other team’s goal, thereby scoring a goal. The players confuse each other by changing their speed or direction unexpectedly. Due to them having the same jersey color, players look almost identical and are frequently possess the ball, which leads to severe occlusions and tracking ambiguities. In such a case, a jersey number must be detected to recognize the player [32][60]. Accurate tracking [33][34][35][36][37][38][39][40][41][42][43][44][61,62,63,64,65,66,67,68,69,70,71,72] by detection [45][46][47][48][73,74,75,76] of multiple soccer players as well as the ball in real-time is a major challenge to evaluate the performance of the players, to find their relative positions at regular intervals, and to link spatiotemporal data to extract trajectories. The systems which evaluate the player [49][77] or team performance [50][78] have the potential to understand the game’s aspects, which are not obvious to the human eye. These systems are able to evaluate the activities of players successfully [51][79] such as the distance covered by players, shot detection [52][53][80,81], the number of sprints, player’s position, and their movements [54][55][82,83], the player’s relative position concerning other players, possession [56][84] of the soccer ball and motion/gesture recognition of the referee [57][85], predicting player trajectories for shot situations [58][86]. The generated data can be used to evaluate individual player performance, occlusion handling [59][21] by the detecting position of the player [60][87], action recognition [61][88], predicting and classifying the passes [62][63][64][89,90,91], key event extraction [65][66][67][68][69][70][71][72][73][74][92,93,94,95,96,97,98,99,100,101], tactical performance of the team [75][76][77][78][79][102,103,104,105,106], and analyzing the team’s tactics based on the team formation [80][81][82][107,108,109], along with generating highlights [83][84][85][86][110,111,112,113]. Table 23 summarizes various proposed methodologies to resolve various challenging tasks in soccer with their limitations.
Table 23.
Studies in Soccer.
| Studies in Soccer | ||||
|---|---|---|---|---|
| Ref. | Problem Statement | Proposed Methodology | Precision and Performance Characteristics | Limitations and Remarks |
| [43][71] | Player and ball detection and tracking in soccer. | YOLOv3 and SORT | Methodology achieved tracking accuracy of 93.7% on multiple object tracking accuracy metrics with a detection speed of 23.7 FPS and a tracking speed of 11.3 FPS. | This methodology effectively handles challenging situations, such as partial occlusions, players and the ball reappearing after a few frames, but fails when the players are severely occluded. |
| [44][72] | Player, referee and ball detection and tracking by jersey color recognition in soccer. | DeepPlayerTrack | The model achieved a tracking accuracy of 96% and 60% on MOTA and GMOTA metrics, respectively, with a detection speed of 23 FPS. | The limitation of this method is that, when a player with the same jersey color is occluded, the ID of the player is switched. |
| [49][77] | Tracking soccer players to evaluate the number of goals scored by a player. | Machine Learning and Deep Reinforcement Learning. | Performance of the player tracking model measured in terms of mAP achieved 74.6%. | The method failed to track the ball at critical moments such as passing at the beginning and shooting. It also failed to overcome the identity switching problem. |
| [67][94] | Extracting ball events to classify the player’s passing style. | Convolutional Auto-Encoder | The methodology was evaluated in terms of accuracy and achieved 76.5% for 20 players. | Concatenation of the auto-encoder and extreme learning machine techniques will improve classification of the event performance. |
| [74][101] | Detecting events in soccer. | Variational Auto- encoder and EfficientNet | Achieved an F1-score of 95.2% event images and recall of 51.2% on images not related to soccer at a threshold value of 0.50. | The deep extreme learning machine technique which employs the auto-encoder technique may enhance the event detection accuracy. |
| [54][82] | Action spotting soccer video. | YOLO-like encoder | The algorithm achieved an mAP of 62.5%. | - |
| [50][78] | Team performance analysis in soccer | SVM | Prediction models achieved an overall accuracy of 75.2% in predicting the correct segmental and the outcome of the likelihood of the team making a successful attempt to score a goal on the used dataset. | The proposed model failed in identifying the players that are more frequently involved in match events that end with an attempt at scoring i.e., a `SHOT’ at goal, which may assist sports analysts and team staff to develop strategies suited to an opponent’s playing style. |
| [57][85] | Motion Recognition of assistant referees in soccer | AlexNet, VGGNet-16, ResNet-18, and DenseNet-121 | The proposed algorithm achieved 97.56% accuracy with real-time operations. | Though the proposed algorithm is immune to variations of illuminance caused by weather conditions, it failed in the case of occlusions between referees and players. |
| [79][106] | Predicting the attributes (Loss or Win) in soccer. | ANN | The proposed model predicts 83.3% for the winning case and 72.7% for loss. | - |
| [82][109] | Team tactics estimation in soccer videos. | Deep Extreme Learning Machine (DELM). | The performance of the model is measured on precision, recall, and F1-score and achieved 87.6%, 88%, and 87.8%, respectively. | Team tactics are estimated based on the relationship between tactics of the two teams and ball possession. The method fails to estimate the team formation at the beginning of the game. |
| [61][88] | Action recognition in soccer | CNN-based Gaussian Weighted event-based Action Classifier architecture | Accuracy in terms of F1-score achieved was 52.8% for 6 classes. | By classifying the actions into subtypes, the accuracy of action recognition can be enhanced. |
| [34][62] | Detection and tracking of the ball in soccer videos. | VGG – MCNN | Achieved an accuracy of 87.45%. | It could not detect when the ball moved out of play in the field, in the stands region, or from partial occlusion by players, or when ball color matched the player’s jersey. |
| [68][95] | Automatic event extraction for soccer videos based on multiple cameras. | YOLO | The U-encoder is designed for feature extraction and has better performance in terms of accuracy compared with fixed feature extractors. | To carry out a tactical analysis of the team, player trajectory needs to be analyzed. |
| [52][80] | Shot detection in a football game | MobileNetV2 | The MobileNetV2 method performed better than other feature extractor methods. | Extracting the features with the MobileNetV2 and then using 3D convolution on the extracted features for each frame can improve detection performance. |
| [58][86] | Predicting player trajectories for shot situations | LSTM | Performance is measured in terms of F1-score and achieved 53%. | The model failed to predict the player trajectory in the case of players confusing each other by changing their speed or direction unexpectedly. |
| [81][108] | Analyzing the team formation in soccer and formulating several design goals. | OpenCV is used for back-end visualization. | The formation detection model achieved a max accuracy of 96.8%. | The model is limited to scalability as it cannot be used on high-resolution soccer videos. The results are bounded to a particular match, and it cannot evaluate the tactical schemes across different games. Visualization of real-time team formation is another drawback as it limits the visualization of non-trivial spatial information. By applying state-of-the-art tracking algorithms, one can predominantly improve the performance of tactics analysis. |
| [32][60] | Player recognition with jersey number recognition. | Spatial Constellation + CNN | Achieved an accuracy of 82% by combining Spatial Constellation + CNN models. | The proposed model failed to handle the players that are not visible for certain periods. Predicting the position of invisible players could improve the quality of spatial constellation features. |
| [62][89] | Evaluating and classifying the passes in a football game. | SVM | The proposed model achieves an accuracy of 90.2% during a football match. | To determine the quality of each pass, some factors such as pass execution of player in a particularly difficult situation, the strategic value of the pass, and the riskiness of the pass need to be included. To rate the passes in sequence, it is necessary to consider the sequence of passes during which the player possesses the ball. |
| [56][84] | Detecting dribbling actions and estimating positional data of players in soccer. | Random forest | Achieved an accuracy of 93.3%. | The proposed methodology fails to evaluate the tactical strategies. |
| [76][103] | Team tactics estimation in soccer videos. | SVM | The performance of the methodology is measured in terms of precision, recall, and F1-score and achieved 98%, 97%, and 98%. | The model fails when audiovisual features could not recognize quick changes in the team’s tactics. |
| [66][93] | Analyzing past events in the case of non-obvious insights in soccer. | k-NN, SVM | To extract the features of pass location, they used heatmap generation and achieved an accuracy of 87% in the classification task. | By incorporating temporal information, the classification accuracy can be improved and also offers specific insights into situations. |
| [33][61] | Tracking the players in soccer videos. | HOG + SVM | Player detection is evaluated in terms of accuracy and achieved 97.7%. Classification accuracy using k-NN achieved 93% for 15 classes. | - |
| [51][79] | Action classification in soccer videos | LSTM + RNN | The model achieves a classification rate of 92% on four types of activities. | By extracting the features of various activities, the accuracy of the classification rate can be improved. |
3. Cricket
In many aspects of cricket as well, computer vision techniques can effectively replace manual analysis. A cricket match has many observable elements including batting shots [87][88][89][90][91][92][93][94][114,115,116,117,118,119,120,121], bowling performance [95][96][97][98][99][100][122,123,124,125,126,127], number of runs or score depending on ball movement, detecting and estimating the trajectory of the ball [101][128], decision making on placement of players’ feet [102][129], outcome classification to generate commentary [103][104][130,131], detecting umpire decision [105][106][132,133]. Predicting an individual cricketer’s performance [107][108][134,135] based upon his past record can be critical in team member selection at international competitions. Such process are highly subjective and usually require much expertise and negotiation decision-making. By predicting the results of cricket matches [109][110][111][112][113][136,137,138,139,140] such as the toss decision, home ground, player fitness, player performance criteria [114][141], and other dynamic strategies the winner can be estimated. The video summarization process gives a compact version of the original video for ease in managing the interesting video contents. Moreover, the video summarization methods capture the interest of the viewer by capturing exciting events from the original video [115][116][142,143]. Table 34 summarizes various proposed methodologies with their limitations to resolve various application issues in cricket.
Table 34.
Studies in Cricket.
| Studies in Cricket | ||||
|---|---|---|---|---|
| Refs. | Problem Statement | Proposed Methodology | Precision and Performance Characteristics | Limitations and Remarks |
| [90][117] | Shot classification in cricket. | CNN—Gated Recurrent Unit | It is evaluated in terms of precision, recall, and F1-score and achieved 93.40%, 93.10%, and 93% for 10 types of shots. | By incorporating unorthodox shots which are played in t20 in the dataset may improve the testing accuracy. |
| [98][125] | Detecting the action of the bowler in cricket. | VGG16-CNN | It was evaluated in terms of precision, recall, and F1-score and the maximum average accuracy achieved is 98.6% for 13 classes (13 types of bowling actions). | Training the model with the dataset of wrong actions can improve detection accuracy. |
| [102][129][136][162] | Tennis swing classification. | SVM, Neural Network, K-NN, Random Forest, Decision Tree | Maximum classification accuracy of 99.72% achieved using NN with a Recall of 1. The second-highest classification accuracy of 99.44% was achieved using K-NN with a recall of 0.98. | If the play styles of the players are different but the patterns are the same, in that case, models failed to classify the current swing direction. |
| [130][156] | Player activity recognition in a tennis game. | Long Short Term Memory (LSTM) | The average accuracy of player activity recognition based on the historical LSTM model was 0.95, and that of the typical LSTM model was 0.70. | The model lacks real-time learning ability and requires a large computing time at the training stage. The model also lacks online learning ability. |
| [121][147] | Automatic detection and classification of change of direction from player tracking data in a tennis game. | Random Forest Algorithm | Among all the proposed methods, model 1 had the highest F1-score of 0.801, as well as the smallest rate of false-negative classification (3.4%) and average accuracy of 80.2% | In the case of non-linear regression analysis, the classification performance of the proposed model is not up to the mark. |
| [127][153] | Prediction of shot location and type of shot in a tennis game. | Generative Adversarial Network (GAN) (Semi-Supervised Model) | The performance factor is measured based on the minimum distance recorded between predicted and ground truth shot location. | The performance of the model deviates from the different play styles as it is trained on the limited player dataset. |
| [133][159] | Analyzing individual tennis matches by capturing spatio-temporal data for player and ball movements. | For data extraction, a player and ball tracking system such as HawkEye is used. | Generation of 1-D space charts for patterns and point outcomes to analyze the player activity. | The performance of the model deviates from different matches, as it was trained only on limited tennis matches. |
| [131][157] | Action recognition in tennis | 3-Layered LSTM | The classification accuracies are as follows: Improves from 84.10 to 88.16% for players of mixed abilities. Improves from 81.23 to 84.33% for amateurs and from 87.82 to 89.42% for professionals, when trained using the entire dataset. | The detection accuracy can be increased by incorporating spatio-temporal data and combining the action recognition data with statistical data. |
| [135][161] | Shot prediction and player behavior analysis in tennis | For data extraction, player and ball tracking systems such as HawkEye are used and a Dynamic Bayesian Network for shot prediction is used. | By combining factors (Outside, Left Top, Right Top, Right Bottom) together, speed, start location, the player movement assessment achieved better results of 74% AUC. | As the model is trained on limited data (only elite players), it cannot be performed on ordinary players across multiple tournaments. |
| [122][148] | Ball tracking in tennis | Two-Layered Data Association | Evaluation results in terms of precision, recall, F1-score are 84.39%, 75.81%, 79.87% for Australian open tennis matchwa and 82.34%, 67.01%, 73.89% for U.S open tennis matches. | The proposed method cannot handle multi-object tracking and it is possible to integrate audio information to facilitate high-level analysis of the game. |
| [134][160] | Highlight extraction from rocket sports videos based on human behavior analysis. | SVM | The proposed algorithm achieved an accuracy of 90.7% for tennis videos and 87.6% for badminton videos. | The proposed algorithm fails to recognize the player, as the player is a deformable object of which the limbs perform free movement during action recognition. |
5. Volleyball
In volleyball, two teams of six players each are placed on either side of a net. Each team attempts to ground a ball on the opposite team’s court and to score points under the defined rules. So, detecting and analyzing the player activities [137][138][139][163,164,165], detecting play patterns and classifying tactical behaviors [140][141][142][143][166,167,168,169], predicting league standings [144][170], detecting and classifying spiking skills [145][146][171,172], estimating the pose of the player [147][173], tracking the player [148][174], tracking the ball [149][175], etc., are the major aspects of volleyball analysis. Predicting the ball trajectory [31][59] in a volleyball game by observing the motion of the setter player has been conducted. Table 56 summarizes various proposed methodologies to resolve various challenging tasks in volleyball sport with their limitations.
Table 56.
Studies in volleyball.
| Studies in Volleyball | ||||
|---|---|---|---|---|
| Ref. | Problem Statement | Proposed Methodology | Precision and Performance Characteristics | Limitations and Remarks |
| [147][173] | Group activity recognition by tracking players. | CNN + Bi-LSTM | The model achieved an accuracy of 93.9%. | The model fails to track the players if the video is taken from a dynamic camera. Temporal action localization can improve the accuracy of tracking the players in severe occlusion conditions. |
| [148][174] | Recognizing and classifying player’s behavior. | SVM | The achieved recognition rate was 98% for 349 correct samples. | - |
| [141][167 | ||||
Ice hockey is another similar game to field hockey, with two teams with six players each, wearing skates and competing on an ice rink. All players aim to propel a vulcanized rubber disk, the puck, past a goal line and into a net guarded by a goaltender. Ice hockey is gaining huge popularity on international platforms due to its speed and frequent physical contact. So, detecting/tracking the player [158][159][160][184,185,186], estimating the pose of the player [161][187], classifying and tracking with different identification the players of the same team or different teams, tracking the ice hockey puck [162][188], and classification of puck possession events [163][189] are the major aspects of the ice hockey sport. Table 67 summarizes various proposed methodologies to resolve various challenging tasks in hockey/ice hockey with their limitations.
Table 67.
Studies in hockey.
| Studies in Hockey | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Ref. | Problem Statement | Proposed Methodology | Precision and Performance Characteristics | Limitations and Remarks | |||||||
| [150][176] | Detecting the player in hockey. | SVM, Faster RCNN, SSD, YOLO | HD+SVM achieved the best results in terms of accuracy, recall, and F1-score with values of 77.24%, 69.23%, and 73.02%. | The model failed to detect the players in occlusion conditions. | |||||||
| [162][188] | Localizing puck Position and Event recognition. | Faster RCNN | Evaluated in terms of AUC and achieved 73.1%. | Replacing the detection method with the YOLO series can improve the performance. | |||||||
| Movement detection of the batsman in cricket. | Deep-LSTM | The model was evaluated in terms of mean square error and achieved a minimum error of 1.107. | - | ||||||||
| ] | Classification of tactical behaviors in beach volleyball. | ||||||||||
| [155][181] | RNN + GRU | The model achieves better classification results as prediction accuracies range from 37% for forecasting the attack and direction to 60% for the prediction of success. | By employing a state-of-the-art method and training on a proper dataset that has continuous positional data, it is possible to predict tactics behavior and set/match outcomes. | [ | |||||||
| Identification of players in hockey. | ResNet + LSTM | Achieves player identification accuracy of over 87% on the split dataset. | Some of the jersey number classes such as 1 to 4 are incorrectly predicted. The diagonal numbers from 1 to 100 are falsely classified due to the small number of training examples. | 115][116][142, | [143] | Cricket video summarization. | 149][175] | Motion estimation for volleyball | Machine Vision and Classical particle filter. | Tracking accuracy is 89% | Replacing methods with deep learning algorithms gives better results. |
| [151][177] | Activity recognition in a hockey game. | Gated Recurrent Neural Network + Hybrid Rotation Forest-Deep Belief Networks YOLO | The methodology was evaluated in terms of precision, F1-score, accuracy and achieved 96.82%, 94.83%, and 96.32% for four classes. YOLO is evaluated on precision, recall, and F1-score and achieved 97.1%, 94.4%, and 95.7% for 8 classes. | Decision tree classifier performance is low due to the existence of a huge number of trees. Therefore, a small change in the decision tree may improve the prediction accuracy. Extreme Learning Machines have faced the problem of overfitting, which can be overcome by removing duplicate data in the dataset. | |||||||
| LSTM | The proposed model recognizes the activities such as free hits, goals, penalties corners, and long corners with an accuracy of 98%. | As the proposed model is focused on spatial features, it does not recognize activities such as free hits and long corners as they appear as similar patterns. By including temporal features and incorporating LSTM into the model, the model is robust to performance accuracy. | [107][134] | Prediction of individual player performance in cricket | Efficient Machine Learning Techniques | The proposed algorithm achieves a classification accuracy of 93.73% which is good compared with traditional classification algorithms. | [Replacing machine learning techniques with deep learning techniques may improve the performance in prediction even in the case of different environmental conditions. | ||||
| 142][168] | Assessing the use of Inertial Measurement Units in the recognition of different volleyball actions. | KNN, Naïve Bayes, SVM | Unweighted Average Recall of 86.87% | By incorporating different frequency domain features, the performance factor can be improved. | Pose estimation and temporal-based action recognition in hockey. | VGG19 + LiteFlowNet + CNN | A novel approach was designed and achieved an accuracy of 85% for action recognition. | The architecture is not robust to abrupt changes in the video, e.g., it fails to predict hockey sticks. Activities such as a goal being scored, or puck location, are not recognized. | [87 | ||
| [154][ | ][114] | Classification of different batting shots in cricket. | CNN | The average classification in terms of precession is 0.80, Recall is 0.79 and F1-score is 0.79. | [To improve the accuracy of classification, a deep learning algorithm has to be replaced with a better neural network. | ||||||
| 31][59] | Predicting the ball trajectory in a volleyball game by observing the the motion of the setter player. | Neural Network | The proposed method predicts 0.3 s in advance of the trajectory of the volleyball based on the motion of the setter player. | In the case of predicting the 3D body position data, the method records a large error. This can be overcome by training properly annotated large data on state-of-art-methods. | |||||||
| [161][187] | Action recognition in ice hockey using a player pose sequence. | CNN+LSTM | The performance of the model is better in similar classes such as passing and shooting. It achieved 90% parameter reduction and 80% floating-point reduction on the HARPET dataset. | [103][ | |||||||
| 180] | As the number of hidden units to LSTM increases, the number of parameters also increases, which leads to overfitting and low test accuracy. | 130] | [Outcome classification task to create automatic commentary generation. | 138CNN + LSTM | Maximum of 85% of training accuracy and 74% validation accuracy | Due to the unavailability of the standard dataset for the ball by ball outcome classification in cricket, the accuracy is not up to mark. In addition, better accuracy leads to automatic commentary generation in sports. | |||||
| ][164] | Activity recognition in beach volleyball | Deep Convolutional LSTM | The approach achieved a classification accuracy of 83.2%, which is superior compared with other classification algorithms. | Instead of using wearable devices, computer vision architectures can be used to classify the activities of the players in volleyball. | |||||||
| [152][178] | Human activity recognition in hockey. | CNN+LSTM | An F1-score of 67% was calculated for action recognition on the multi-labeled imbalanced dataset. | The performance of the model is poor because of the improper imbalanced dataset. | [105][132] | ||||||
| [144 | Detecting the third umpire decision and an automated scoring system in a cricket game. | ][170CNN + Inception V3 | ]It holds 94% accuracy in the Deep Conventional Neural Network (DCNN) and 100% in Inception V3 for the classification of umpire signals to automate the scoring system of cricket. | To build an automated umpiring system based on computer vision application and artificial intelligence, the results obtained in this paper are more than enough. | |||||||
| Volleyball skills and tactics analysis | ANN | Evaluated in terms of Average Relative Error for 10 samples and achieved 0.69%. | - | ||||||||
| [153][179] | Player action recognition in an ice hockey game | CNN | The accuracy of the actions recognized in a hockey game is 65% and when similar actions are merged accuracy rises to 78%. | Pose estimation problems due to severe occlusions when motions blur due to the speed of the game and also due to lack of a proper dataset to train models, all causing low accuracy. | [106][133] | ||||||
| [139] | Classification of cricket bowlers based on their bowling actions. | [CNN | The test set accuracy of the model is 93.3% which demonstrates its classification ability. | The model lacks data for detecting spin bowlers. As the dataset is confined to left-arm bowlers, the model misclassifies the right-arm bowlers. | |||||||
| 165] | Group activity recognition in a volleyball game | LSTM | Group activity recognition of accuracy of the the proposed model in volleyball is 51.1%. | [88][115] | Recognition of various batting shots in a cricket game | Deep-CNN | The proposed models can recognize a shot being played with 90% accuracy. | As the model is dependent on the frame per second of the video, it fails to recognize when the frames per second increases. | |||
| [104][131] | Automatic highlight generation in the game of cricket. | CNN + SVM | Mean Average Precision of 72.31% | The proposed method cannot clear metrics to evaluate the false positives in highlights. | |||||||
| [106][133] | Umpire pose detection and classification in cricket. | SVM | VGG19-Fc2 Player testing accuracy of 78.21% | Classification and summarization techniques can minimize false positives and false negatives. | |||||||
| [89][116] | Activity recognition for quality assessment of batting shots. | Decision Trees, k-Nearest Neighbours, and SVM. | The proposed method identifies 20 classes of batting shots with an average F1-score of 88% based on the recorded movement of data. | To assess the player’s batting caliber, certain aspects of batting also need to be considered, i.e., the position of the batsman before playing a shot and the method of batting shots for a particular bowling type can be modeled. | |||||||
| [109][110][136,137] | Predicting the outcome of the cricket match. | k-NN, Naïve Bayesian, SVM, and Random Forest | Achieved an accuracy of 71% upon the statistics of 366 matches. | Imbalance in the dataset is one of the causes which produces lower accuracy. Deep learning methodologies may give promising results by training with a dataset that included added features. | |||||||
| [97][124] | Performance analysis of the bowler. | Multiple regression | Variation in ball speed has a feeble significance in influencing the bowling performance (the p-value being 0.069). The variance ratio of the regression equation to that of the residuals (F-value) is given as 3.394 with a corresponding p-value of 0.015. | - | |||||||
| [108][135] | Predicting the performance of the player. | Multilayer perceptron Neural Network | The model achieves an accuracy of 77% on batting performance and 63% on bowling performance. | - | |||||||
4. Tennis
Worldwide, Tennis has experienced gain a huge popularity. This game need a meticulous analysis to reducing human errors and extracting several statistics from the game’s visual feed. Automated ball and player tracking belongs to such class of systems that requires sophisticated algorithms for analysis. The primary data for tennis are obtained from ball and player tracking systems, such as HawkEye [117][118][144,145] and TennisSense [119][120][28,146]. The data from these systems can be used to detect and track the ball/player [121][122][123][124][147,148,149,150], visualizing the overall tennis match [125][126][151,152] and predicting trajectories of ball landing positions [127][128][129][153,154,155], player activity recognition [130][131][132][156,157,158], analyzing the movements of the player and ball [133][159], analyzing the player behavior [134][160] and predicting the next shot movement [135][161] and real-time tennis swing classification [136][162]. Table 45 summarizes various proposed methodologies to resolve various challenging tasks in tennis with their limitations.
Table 45.
Studies in Tennis.
| Studies in Tennis | ||||
|---|---|---|---|---|
| Ref. | Problem Statement | Proposed Methodology | Precision and Performance Characteristics | Limitations and Remarks |
| [118][145] | Monitoring and Analyzing tactics of tennis players. | YOLOv3 | The model achieved an mAP of 90% with 13 FPS on high-resolution images. | Using a lightweight backbone for detection, modules can improve the processing speed. |
| [132][158] | Player action recognition in tennis. | Temporal Deep Belief Network (Unsupervised Learning Model) | The accuracy of the recognition rate is 94.72% | If two different movements are similar, then the model fails to recognize the current action. |
| The performance of architecture is poor because of the lack of hierarchical considerations of the individual and group activity dataset. | ||||
6. Hockey/Ice Hockey
Hockey, also known as Field hockey, is an outdoor game played between two teams of 11 players each. These players use sticks that are curved at the striking end to hit a small and hard ball into their opponent’s goal post. So, detecting [150][176] and tracking the player/hockey ball, recognizing the actions of the player [151][152][153][177,178,179], estimating the pose of the player [154][180], classifying and tracking the players of the same team or different teams [155][181], referee gesture analysis [
7. Badminton
Badminton is one of the most popular racket sports, which includes tactics, techniques, and precise execution movements. To improve the performance of the player, technology plays a key role in optimizing the training of players; technology determines the movements of the player [164][190] during training and game situations such as with action recognition [165][166][167][191,192,193], analyzing the performance of player [168][194], detecting and tracking the shuttlecock [169][170][171][195,196,197]. Table 78 summarizes various proposed methodologies to resolve various challenging tasks in badminton with their limitations.
Table 78.
Studies in badminton.
| Studies in Badminton | ||||
|---|---|---|---|---|
| Ref. | Problem Statement | Proposed Methodology | Precision and Performance Characteristics | Limitations and Remarks |
| [169][195] | Shuttlecock detection problem of a badminton robot. | Tiny YOLOv2 and YOLOv3 | Results show that, compared with state-of-art methods, the proposed networks achieved good accuracy with efficient computation. | The proposed method fails to detect different environmental conditions. As it uses the binocular camera to detect a 2D shuttlecock, it cannot detect the 3D shuttlecock trajectory. |
| [165][191] | Automated badminton player action recognition in badminton games. | AlexNet+CNN, GoogleNet+CNN and SVM | Recognition of badminton actions by the linear SVM classifier for both AlexNet and GoogleNet using local and global extractor methods is 82 and 85.7%. | The architecture can be improved by fine-tuning in an end-to-end manner with a larger dataset on features extracted at different fully connected layers. |
| [166][192] | Badminton activity recognition | CNN | Nine different activities were distinguished: seven badminton strokes, displacement, and moments of rest. With accelerometer data, accurate estimation was conducted using CNN with 86% precision. Accuracy is raised to 99% when gyroscope data are combined with accelerometer data. | Computer vision techniques can be employed instead of sensors. |
| [167][193] | Classification of badminton match images to recognize the different actions were conducted by the athletes. | AlexNet, GoogleNet, VGG-19 + CNN | Significantly, the GoogleNet model has the highest accuracy compared to other models in which only two-hit actions were falsely classified as non-hit actions. | The proposed method classifies the hit and non-hit actions and it can be improved by classifying more actions in various sports. |
| [170][196] | Tracking shuttlecocks in badminton | An AdaBoost algorithm which can be trained using the OpenCV Library. | The performance of the proposed algorithm was evaluated based on precision and it achieved an average precision accuracy of 94.52% with 10.65 fps. | The accuracy of tracking shuttlecocks is enhanced by replacing state-of-the-art AI algorithms. |
| [164][190] | Tactical movement classification in badminton | k-Nearest Neighbor | The average accuracy of player position detection is 96.03 and 97.09% on two halves of a badminton court. | The unique properties of application such as the length of frequent trajectories or the dimensions of the vector space may improve classification performance. |
8. Miscellaneous
Player detection and tracking is the major requirement in athletic sports such as running, swimming [172][173][198,199], and cycling. In sports such as table tennis [174][200], squash [175][176][201,202], and golf [177][203], ball detection and tracking and player pose detection [178][204] are challenging tasks. In ball-centric sports such as rugby, American football, handball, baseball, ball/player detection [179][180][181][182][183][184][185][205,206,207,208,209,210,211] and tracking [186][187][188][189][190][191][192][193][194][195][212,213,214,215,216,217,218,219,220,221], analyzing the action of the player [196][197][198][199][200][201][202][23,222,223,224,225,226,227], event detection and classification [203][204][205][206][207][228,229,230,231,232], performance analysis of player [208][209][210][233,234,235], referee identification and gesture recognition are the major challenging tasks. Video highlight generation is a subclass of video summarization [211][212][213][214][236,237,238,239] which may be viewed as a subclass of sports video analysis. Table 89 summarizes various proposed methodologies to resolve various challenging tasks in various sports with their limitations.
Table 89.
Studies in various sports.
| Studies in Various Sports | ||||
|---|---|---|---|---|
| Ref. | Problem Statement | Proposed Methodology | Precision and Performance Characteristics | Limitations and Remarks |
| [185][211] | Beach sports image recognition and classification. | CNN | The model achieved a recognition accuracy of 91%. | Lightweight networks of deep learning algorithms can improve the recognition accuracy and can also be implemented in real-time scenarios. |
| [173][199] | Motion image segmentation in the sport of swimming | GDA + SVM | The performance of the Symmetric Difference Algorithm was measured in terms of recall and achieved 76.2%. | Using advanced optimization techniques such as Cosine Annealing Schedulers with deep learning algorithms may improve the performance. |
| [174][200] | Identifying and recognizing wrong strokes in table tennis. | k-NN, SVM, Naïve Bayes | Performs various ML algorithms and achieves an accuracy of 69.93% using the Naïve Bayes algorithm. | A standard dataset can improve the accuracy of recognizing the wrong strokes in table tennis. |
| [186][212] | Multi-player tracking in sports | Cascade Mask R-CNN | The proposed Deep Player Identification method studies the patterns of jersey number, team class, and pose-guided partial feature. To handle player identity switching, the method correlates the coefficients of player ID in the K-shortest path with ID. The proposed framework achieves state-of-art performance. | When compared with existing methods, the computation cost is higher and can be considered a major drawback of the proposed framework. To refine 2D detection, temporal information needs to be considered and can be transferred to tracking against a real-time performance such as soccer, basketball, etc. |
| [189][215] | Individual player tracking in sports events. | Deep Neural Network | Achieved an Area Under Curve (AUC) of 66% | Tracking by jersey number recognition may increase the performance of the model. |
| [178][204] | Skelton-based key pose recognition and classification in sports | Boltzmann machine+CNN Deep Boltzmann machine + RNN | The proposed architecture successfully analyses feature extraction, motion attitude model, motion detection, and behavior recognition of sports postures. | The architecture is bound to individual-oriented sports and can be further implemented on group-based sports, in case of challenges such as severe occlusion, misdetection due to failure in blob detection in object tracking. |
| [199][224] | Human action recognition and classification in sports | VGG 16 + RNN | The proposed method achieved an accuracy of 92% for ten types of sports classification. | The model fails in the case of scaling up the dataset for larger classification which shows ambiguity between players and similar environmental conditions. Football, Hockey; Tennis, Badminton; Skiing, Snowboarding; these pairs of classes have similar environmental features; thus, it is only possible to separate them based on relevant actions which can be achieved by state-of-the-art methods. |
| [212][237] | Replay and key event detection for sports video summarization | Extreme Learning Machine (ELM) | The framework is evaluated on a dataset that consists of 20 videos of four different sports categories. It achieves an average accuracy of 95.8%, which illustrates the significance of the method in terms of key-event and replay detection for video summarization. | The performance of the proposed method drops in the case of the absence of a gradual transition of a replay segment. It can be extended by incorporating artificial intelligence techniques. |
| [203][228] | Event detection in sports videos for unstructured environments with arbitrary camera angles. | Mask RCNN + LSTM | The proposed method is accurate in unsupervised player extraction, which is used for precise temporal segmentation of rally scenes. It is also robust to noise in the form of camera shaking and occlusions. | It can be extended to doubles games with fine-grained action recognition for detecting various kinds of shots in an unstructured video and it can be extended to analyze videos of games such as cricket, soccer, etc. |
| [204][229] | Human motion quality assessment in complex motion scenarios. | 3-Dimensional CNN | Achieved an accuracy of 81% on the MS-COCO dataset. | Instead of the Stochastic Gradient Descent technique for learning rate, using the Cosine annealing scheduler technique may improve the performance. |
| [187][213] | Court detection using markers, player detection, and tracking using a drone. | Template Matching + Particle Filter | The proposed method achieves better accuracy (94%) in the case of two overlapping players | As the overlapping of players, increases the accuracy of detection and tracking decreases due to similar features of players on the same team. The method uses a template matching algorithm, which can be replaced with a deep learning-based state-of-art algorithm to acquire better results. |
| [188][214] | Target tracking theory and analyses its advantages in video tracking. | Mean Shift + Particle Filter | Achieves better tracking accuracy compared to existing algorithms such as TMS and CMS algorithms. | If the target scales change then the tracking of players fails due to the unchanged window of the mean-shift algorithm. Furthermore, it cannot track objects which are similar to the background color. The accuracy of tracking players can be improved by replacing them with artificial intelligence algorithms. |
| [211][236] | Automatically generating a summary of sports video. | 2D CNN + LSTM | Describes a novel method for automatic summarization of user-generated sports videos and demonstrated the results for Japanese fencing videos. | The architecture can be improved by fine-tuning in an end-to-end manner with a larger dataset for illustrating potential performance and also to evaluate in the context of a wider variety of sports. |
| [202][227] | Action Recognition and classification | SVM | Achieved an accuracy of 59.47% on the HMDB 51 dataset. | In cases where the object takes up most of the frame, the human detector cannot completely cover the body of the object. This leads to the system missing movements of body parts such as hands and arms. In addition, recognition of similar movements is a challenge for this architecture. |
9. Overview of Machine Learning/Deep Learning Techniques
There are multiple ways to classify, detect, and track objects to analyze the semantic levels involved in various sports. They pave the way for player localization, jersey number recognition, event classification and trajectory forecasting of the ball in a sports video with a much better interpretation of an image as a whole.
The selected AI algorithm is better if it is tested and benchmarked on different data. To evaluate the robustness of AI algorithms, some metrics are required which measure the performance of particular AI algorithms to enable better selection. Figure 18 depicts the road map of the machine learning algorithms’ general information, methods, and evaluation criteria for a particular task and required libraries/tools for training the model. Figure 29 depicts the roadmap of the deep learning algorithm selection, training, and evaluation criteria for a particular task and required libraries/tools for training the model. Figure 310 shows taxonomy of various deep learning techniques of classification [215][216][217][218][219][220][240,241,242,243,244,245], detection [221][246] and prediction [222][223][224][247,248,249] algorithms, unsupervised learning [225][226][250,251], tracking [227][228][229][230][231][232][233][234][235][236][252,253,254,255,256,257,258,259,260,261], and trajectory prediction [237][238][239][240][241][242][243][244][262,263,264,265,266,267,268,269]. Since various tasks in sports such as classification/detection, tracking, and trajectory prediction show great advantages in various sports. A bi-layered parallel training architecture in distributed computing environments was introduced in [245][270], which discusses the time-consuming training process of large-scale deep learning algorithms.
Figure 18.
Block diagram of the road map to machine learning architecture selection and training.
Figure 29.
Block diagram of the road map to deep learning architecture selection and training.
Figure 310.
Overview of deep learning algorithms of classification/detection, tracking and trajectory prediction.
