Augmenting Sports Videos with VisCommentator

Machine Learning Library and Code

1. BodyPix2

We use BodyPix2 from google to detect and segment players from a video. BodyPix2 is devloped based on Tensorflow.js and can be ran in NodeJs environments. The hyper-parameters we used are:

// for bodyPix.load
{
    "architecture": "ResNet50",
    "outputStride": 16,
    "quantBytes": 4
}
 // for net.segmentPerson
{
    "flipHorizontal": false,
    "internalResolution": "full",
    "segmentationThreshold": 0.7,
    "maxDetections": 10,
    "scoreThreshold": 0.2,
    "nmsRadius": 20,
    "minKeypointScore": 0.3,
    "refineSteps": 10
}
          

2. Event Recognition

We use spatial temporal graph ConvNet (ST-GCN) [1] to recongize player event such as the technique a player used in one stroke.

2.1. Input & output

According to the pipeline porposed in ST-GCN, we first collect the table tennis video with high resolution (1080p) and frame rate (50fps). Then we use the pose estimation algorithm to retrieve the body joints from each frame, and formulate the spatial graph where the node is the joint and edge is the connection between joints of a body and joints between adjacent frames. Finally the spatial graph is used as the input for the ST-GCN.

The output of the ST-GCN is the standard SoftMax classifier that identifies the graph into corresponding technique category.

2.2. Training

We kept the the network structure the same as it introduced in the paper [1]. To gather the training data, we collected thousands of strokes from hundreds of world tournament games happened during 2016-2018. These data are manually identified from videos and formulated into structured json file. A sample file fragment is as follows:

To train the ST-GCN, we kept the number of records (video fragments) in each technique equally to make sure the scale of joints from different categories are almost the same. To this end, we first further devided the records of each technique into two categories: the player face to the screen and the player back to the screen. Then, we grouped the techniques with little records into "other" category, and simply discarded extra records to maintain consistent record numbers for each category. Finally, we got seven techniques (i.e. topspin, reverse, push, short, pendulum, attack, others), 14 categories with 4375 records each.

3. Causality

We use FGES (Fast Greedy Equivalence Search) [2] to extract the recommended data that used for future purpose. The toolkit can be found in [3].

3.1. Input

To generate a causal graph, we group the table tennis data into a table, where each row represents a rally and each column represents a technique. Each cell records the frequency of corresponding technique used in that rally.

Input Table Sample

3.2. Output