A) People tracking

Visual object tracking is one of the fundamental problems in video analysis and understanding. Given the bounding box of a target object in the first frame of a video, a tracker is expected to locate the target object in all subsequent frames. Single object tracking can be typically considered a joint detection and tracking problem as it is essentially the detection of the same target object in every subsequent frame. The greatest challenge is to fulfill the simultaneous, but somewhat conflicting requirements on robustness and discrimination power [Reference Wang, Luo, Xiong and Zeng7]. The robustness requires a tracker not to lose tracking when the appearance of the target changes due to illumination, motion, view angle, or object deformation. Meanwhile, a tracker is expected to have the capability to discriminate the target object from a cluttered background or similar surrounding objects. Both requirements traditionally need to be handled through online training to achieve the adaptability.

Since 2015, more and more works based on CNN have emerged. While deep features introduce the speed limitation to online training, their strong representational power opens-up a possibility to completely remove online training. The pioneering work along this line is SiamFC [Reference Bertinetto, Valmadre, Henriques, Vedaldi and Torr8]. SiamFC employs offline-trained Siamese CNNs to extract features, and then uses a simple cross-correlation layer to perform a dense and efficient sliding-window-based search. As a result, SiamFC can operate at 86fps on GPU. There are a great number of follow-up works, such as SA-Siam [Reference He, Luo, Tian and Zeng9] and SiamRPN [Reference Li, Yan, Wu, Zhu and Hu10]. SA-Siam adopts a two-branch network to encode images into two embedding spaces, one for semantic similarity (robustness) and the other for appearance similarity (discriminativeness). SiamRPN consists of a Siamese subnetwork for feature extraction and a region proposal subnetwork for similarity matching and box regression.

While most object tracking works use a single-stage design, a two-stage SiamFC-based network is proposed in [Reference Wang, Luo, Xiong and Zeng7], aiming to address both the robustness and discriminativeness requirements. The two stages are the coarse matching stage (i.e. a proposal stage) which focuses on enhancing the robustness, and the fine matching stage which focuses on improving the discrimination power by replacing the cross-correlation layer with a more powerful distance learning subnetwork. The resulting tracker achieves superior performance on major benchmark datasets, with an impressive inference speed of 120fps on an NVIDIA P100 GPU. Figure 2 shows an accuracy-speed trade-off of top-performing trackers on the OTB-100 benchmark [Reference Wu, Lim and Yang11]. It can be observed that significant progress has been made for single-object tracking in the past few years. These can be readily applied to single-person tracking.

Fig. 2. Accuracy-speed trade-off of top-performing trackers on the OTB-100 benchmark. The speed axis is logarithmic. Reproduced from Fig. 8 of [Reference Wang, Luo, Xiong and Zeng7]. Please refer to [Reference Wang, Luo, Xiong and Zeng7] for the notations of different trackers. “Ours” refers to the SPM-Tracker [Reference Wang, Luo, Xiong and Zeng7].

In many practical scenarios, it is required to track multiple people simultaneously. It is generally not efficient to treat multiple-person tracking as multiple separate single-person tracking tasks. Recent works on multi-person tracking focus on the tracking-by-detection approach, i.e. detecting objects (of the same target category, i.e. person) using a general object/person detector in individual frames and then linking detections across frames. These include various strategies such as importance sampling and particle filtering for state propagation in a Bayesian framework [Reference Giebel, Gavrila and Schnorr12], linking short tracks over a long duration, e.g. using the Hungarian algorithm for the optimal assignment [Reference Perera, Srinivas, Hoogs, Brooksby and Wensheng13], and greedy Dynamic Programming in which trajectories are estimated one after another [Reference Fleuret, Berclaz, Lengagne and Fua14]. To improve robustness to wrong identity assignment, recent research has focused on linking detections over a larger time duration using various optimization schemes. One common formulation to address multi-person tracking is based on constrained flow optimization which can be solved using the k-shortest paths algorithm [Reference Berclaz, Fleuret, Turetken and Fua15]. Graph-based minimum cost multicut formulation has also been proposed [Reference Ristani and Tomasi16, Reference Tang, Andres, Andriluka and Schiele17]. Note that the tracking-by-detection approach separates tracking from detection, therefore may not be the most efficient approach. For example, a general object detector may miss detecting some object instances in some frames, which otherwise could be tracked if a single-object tracker is used for that specific object instance. We believe more efforts should be made to investigate the approach of joint detection and tracking, which has been extensively studied for single-object tracking, for joint multiple-object/people tracking. The spatial and temporal relationships between multiple objects/people should be better exploited. There is also a trade-off between complexity and accuracy that should be optimized. Tracking over a long period of time is typically very challenging. Long-term tracking usually results in intermediate short-term tracklets, and requires linking/matching tracklets over time, e.g. through object re-identification (re-ID) techniques.

While general object tracking can be readily applied to people tracking, there are dedicated tracking technologies designed for people tracking. For example, a detector can be designed specifically for people to handle occlusion and body deformation [Reference Tang, Andriluka, Milan, Schindler, Roth and Schiele18]. A minimum cost-lifted multicut formulation was proposed in [Reference Tang, Andriluka, Andres and Schiele19] to introduce additional edges in the graph to incorporate long-range person re-ID information into the tracking formulation. To effectively match hypotheses over longer temporal gaps, new deep architectures for re-ID of people are developed, where holistic deep feature representations and extracted body pose layout are combined. More discussion about person re-ID will be presented in Section III.C.

B) Human pose estimation

Human pose estimation determines the pixel locations of key joints of the human body. It is a key step toward understanding people. It has widespread applications such as human action recognition, motion analysis, activity analysis, and human–computer interaction. Despite many years of research with significant progress made recently, pose estimation remains a very challenging task, mainly due to the large variations in body postures, shapes, capturing views, scales, complex inter-dependency of parts, appearances, quality of images, etc.

The pictorial structure [Reference Yang and Ramanan20] is an early work that defines the deformable configurations by spring-like connections between pairs of parts to model complex joint relations. Subsequent works [Reference Chen and Yuille21, Reference Yang, Ouyang, Li and Wang22] extend this idea to CNNs. Many recent works use CNNs to learn feature representations and obtain the locations of the 2D joints or the score maps of the 2D joints [Reference Toshev and Szegedy23–Reference Newell, Yang and Deng25]. Some methods directly employ learned feature representations to regress the 2D joint positions [Reference Toshev and Szegedy23], while a popular way of a joint detection is to estimate a score map for each 2D joint based on fully CNN [Reference Wei, Ramakrishna, Kanade and Sheikh24–Reference Cao, Simon, Wei and Sheikh26]. To efficiently detect the 2D poses of multiple people, [Reference Cao, Simon, Wei and Sheikh26] uses Part Affinity Fields to learn to associate body parts with individuals in the image. The architecture encodes global context, allowing a greedy bottom-up parsing step that maintains high accuracy while achieving real-time performance, irrespective of the number of people in the image. A two-stage normalization scheme, i.e. human body normalization followed by limb normalization, is presented in [Reference Sun, Lan, Xing, Wang, Zeng and Liu27] to make the distribution of the relative joint locations compact, resulting in easier learning of convolutional spatial models and more accurate pose estimation.

In addition to 2D pose estimation discussed above, there have been recent efforts to estimate relative 3D poses from monocular images [Reference Martinez, Hossain, Romero and Little28–Reference Sun, Xiao, Wei, Liang and Wei30], mostly using regression methods. It is shown in [Reference Sun, Xiao, Wei, Liang and Wei30] that a simple integral operation (replacing the “taking-maximum” operation by “taking-expectation”) can relate and unify the heat map representation and joint regression, taking advantages of their individual merits to further improve the performance.

Another practically more important task is to estimate absolute 3D human poses from multiple calibrated cameras [Reference Hartley and Zisserman31]. Many earlier works [Reference Amin, Andriluka, Rohrbach and Schiele32] follow the pipeline of first locating the 2D joints in each camera view and then triangulating them to 3D [Reference Hartley and Zisserman31]. The 3D pose estimation accuracy heavily depends on the accuracy of the 2D joint estimations. In [Reference Qiu, Wang, Wang, Wang and Zeng33], a cross-view feature fusion approach to fuse the multi-view features in order to achieve more robust 2D pose estimation for each view, especially for the occluded joints, is presented. A Recursive Pictorial Structure Model (RPSM) is then presented to estimate 3D poses from multi-view 2D poses. This work sets a new state-of-the-art on multi-view human pose estimation on the benchmark H36M dataset [Reference Ionescu, Papava, Olaru and Sminchisescu34], outperforming prior works (e.g. [Reference Tome, Toso, Agapito and Russell35]) by a large margin (a 50% reduction of the average joint estimation error). This makes high accuracy motion capture without motion sensors or markers very close to reality, and is expected to enable many applications such as low-cost athlete body motion analysis, human tracking and action recognition in retail scenarios, etc.

C) Person re-identification

Person re-ID aims to match a specific person across multiple camera views or in different occasions from the same camera view. It facilitates many important applications, such as cross-camera tracking [Reference Wang36] and the long-term people tracking discussed in Section III.A. This task is very challenging due to large variations of person pose and viewpoint, imperfect person detection, cluttered background, occlusion, and lighting differences, etc. Many of these factors result in spatial misalignment of two matching human bodies, making it one of the key challenges in re-ID.

In recent years, many efforts have been made to alleviate these problems [Reference Varior, Shuai, Lu, Xu and Wang37–Reference Suh, Wang, Tang, Mei and Lee39]. For example, to make the learned features focus on some local details, some works make a straightforward partition of the person image into a few fixed rigid parts (e.g. horizontal stripes) and learn detailed local features [Reference Varior, Shuai, Lu, Xu and Wang37, Reference Cheng, Gong, Zhou, Wang and Zheng40, Reference Wang, Yuan, Chen, Li and Zhou41]. Such partition however cannot align well with the human body parts. Some other works have attempted to use the pose to help localize body parts for learning part-aligned features [Reference Su, Li, Zhang, Xing, Gao and Tian38, Reference Li, Chen, Zhang and Huang42, Reference Zhao43]. This, however, is a very coarse alignment. Even for the same type of parts, there is still spatial misalignment within the parts between images, where the human semantics are different for the same spatial positions. It becomes critical to design an architecture that enables the efficient learning of densely semantically aligned features for re-ID.

A densely semantically aligned person re-ID framework is proposed in [Reference Zhang, Lan, Zeng and Chen44], which enables fine-grained semantic alignment and explores the semantically aligned feature learning. It performs dense semantic alignment of the human body on a canonical space to address the misalignment challenges, where dense (i.e. pixel-wise) semantic pose estimation [Reference Guler, Neverova and Kokkinos45] is leveraged. To address the potential dense pose estimation errors (i.e. robustness issue) and the challenges in handling non-overlapping areas between two matching persons, a powerful joint learning framework is developed to guide the feature learning of one main stream using another (densely semantically aligned, but noisy) stream. State-of-the art performance is achieved on the benchmark datasets. Along the same line, a more elegant framework is proposed in [Reference Jin, Lan, Zeng, Wei and Chen46] that uses an encoder–decoder architecture to guide the feature learning such that the learned features for re-ID is capable of reconstructing, through the decoder, a 3D full body texture image in a canonical semantic space. The learned features used for re-ID are thus view- and pose-invariant. The decoder is used only for model training, without increasing the inference complexity. This approach nicely addresses the issue of visible body inconsistency between matching images, which is not well-addressed in [Reference Zhang, Lan, Zeng and Chen44].

D) Human action recognition

Ultimately, we would like to understand the human activities in the video. Human action recognition thus is a very important but challenging task. There are a large variety of actions, with large or subtle differences. It is therefore important to be able to leverage some attention mechanisms, just like the human brain does, to focus on what really characterize a particular action to differentiate it from other actions. View variation (e.g. among videos taken of the same action) is another significant challenge, which demands view-invariant approaches.

There have been RGB-based approaches [Reference Weinland, Ronfard and Boyerc47–Reference Zhou, Sun, Zha and Zeng53], skeleton-based approaches [Reference Du, Wang and Wang54–Reference Zhang, Lan, Xing, Zeng, Xue and Zheng57], and their combinations [Reference Song, Lan, Xing, Zeng and Liu58] for human action recognition. RGB-based approaches have the advantage of taking into account the appearance information and the context (e.g. background and other objects around humans), but have difficulty differentiating some fine-grained actions (e.g. human poses/motions). Skeleton-based approaches can focus on fine-grained actions of the human body, but lose the context information (e.g. interacting objects). Combining the RGB information and pose information would often provide the best performance [Reference Song, Lan, Xing, Zeng and Liu58].

We focus on skeleton-based approaches here. Earlier neurophysiological study shows that human can recognize action by motions of just a few key points [Reference Johansson59]. In fact, the skeleton can be considered as a high-level abstract representation of the human body, and it reflects some sort of human brain attention. Since there is a sequence of skeleton joint sets over time in the video, one can use a recurrent neural network such as Long Short-Term Memory (LSTM) network [Reference Graves60] to model the temporal dynamics of the joints for different actions. Co-occurrence patterns of joint features are learnt using an end-to-end regularized LSTM network for human action recognition [Reference Zhu55]. Based on the observation that for different actions, the importance levels of different joints are different, both a spatial attention model and a temporal attention model are used to highlight what really characterize a particular action instance [Reference Song, Lan, Xing, Zeng and Liu56]. Figure 3 shows the network architecture. The single-layer LSTM-based spatial attention model assigns different weights to joints of the same frame before applying a baseline action classification network. The single-layer LSTM-based temporal attention model generates a temporal weighting curve to pool action prediction outputs at different time instances. Significant accuracy improvements (up to 6% absolute gains over baseline on a benchmark dataset) have been achieved, signifying the importance of developing attention mechanisms in vision tasks.

Fig. 3. Spatial-temporal attention network. Both attention networks use a one-layer LSTM network.

As mentioned above, a significant challenge for video analysis is view variation, i.e. visual representations of the same event captured from different views would look very different. The human brain has the capability to recognize that they are the same event. We would like the machine to be able to do that as well. A view adaptation sub-network is proposed in [Reference Zhang, Lan, Xing, Zeng, Xue and Zheng57] to address the view-invariant property, by adaptively transforming the input 3D skeleton sequence to a more consistent virtual view before an LSTM-based classification network is applied. Through end-to-end training, this view adaptation sub-network allows the main action classification network to “see” only skeleton sequences of consistent views, disregard their original views. This effectively addresses the view variation problem, resulting in a powerful action classification network that can focus its attention on the action details, as opposed to visual content variation resulting from view variation.

E) Integration

While it is critical to develop individual human-centric vision tasks for video understanding, it is also important to look at them from a system perspective, and understand how to integrate those building blocks, and how they compensate each other.

Depending on the application scenarios, a practical system may integrate some or all of the building blocks. For example, for a workplace safety scenario, people detection and tracking may be sufficient to detect if there is any human activity outside a safety zone. In a more complex retail intelligence scenario where multiple cameras are deployed (see Fig. 4), one may need to detect and track the customers, based on face, body, skeleton, or a combination. Long-term tracking may be necessary and is typically challenging, especially across different cameras, in which case person re-ID becomes critical to link estimated short-term tracklets. With people tracking, heatmap can be created to reflect where in the store customers go/stay most. If more detailed activities of the customers are of interest, one may want to also identify the actions (e.g. picking up an item) of the customers, e.g. by leveraging the estimated pose sequence, or a combination of pose and RGB data. For efficient integration of the building blocks, a mask-RCNN [Reference He, Gkioxari, Dollar and Girshick61] like architecture with a multi-task setting that shares the feature extraction backbone network could be used.

Fig. 4. Illustration of a retail intelligence scenario where multiple cameras are deployed, 3D space is reconstructed, people are detected and tracked, and heatmap (in purple) is generated.

For real-time interactive applications such as video conferencing, the real-time requirement is likely the most significant challenge for many vision tasks. For example, foreground/human body segmentation and background blurring are desirable features to remove background distractions or provide privacy protection. In this case, there is a stringent requirement on the neural network model size (e.g. in the order of 100 K bytes) and speed (in the order of ms per frame). Significant efforts need to be devoted to the model size reduction and speed optimization.