2.1. Gait trajectory prediction

Traditional gait prediction methods represented by controlled oscillators can deal with some simple fixed-period gait prediction problems. Adaptive oscillators can predict cyclic and quasi-cyclic gait sequences., but they cannot predict rapidly changing gaits in time. The traditional gait prediction method has a noticeable delay in predicting rapidly changing gait. This shortcoming limits the application of the traditional method [Reference Xue, Wang, Zhang and Zhang1, Reference Seo, Kim, Park, Cho, Lee, Choi, Lim, Lee and Shim2].

The data-based deep learning method can directly output the trajectory in the future based on the sensor data without an iterative update of parameters, which solves the delay problem. Tanghe et al. and Kang et al. predicted the gait phase [Reference Tanghe, De Groote, Lefeber, De Schutter and Aertbeliën3, Reference Kang, Kunapuli and Young4] based on the deep neural network, but their models had high requirements for the type and quantity of sensors, and the workload of processing data was heavy. High requirements limit the real-world application of their method. It is significant to reduce the type and quantity of sensors as much as possible to reduce the cost of gait prediction.

As an important method for processing sequence information, the long short-term memory (LSTM) model is widely used to predict gait trajectory [Reference Wang, Li, Wang, Yu, Liao and Arifoglu5, Reference Zaroug, Garofolini, Lai, Mudie and Begg6, Reference Zaroug, Lai, Mudie and Begg7, Reference Su and Gutierrez-Farewik8, Reference Liu, Wu, Wang and Chen9]. These models can only predict gait trajectories within a limited time frame. When predicting long-term gait trajectories, the LSTM model they adopted is slow and has a large cumulative error caused by the loop structure of the LSTM model itself.

Karakish et al. and Kang et al. used a CNN-based deep neural network to predict gait [Reference Karakish, Fouz and ELsawaf10, Reference Kang, Molinaro, Duggal, Chen, Kunapuli and Young11], which is faster than the LSTM model. However, the model proposed by Karakish et al. can only predict the gait trajectory in a short time, and the model proposed by Kang et al. relies on user-specific basis data.

The gait trajectory prediction models mentioned above are difficult to predict long-term gait trajectory, and none of them consider the issue of model interpretability. They cannot judge the input’s importance or explain the model’s data basis. The above models only predict gait as a black box and cannot support in-depth gait research.

2.2. Long-term series forecasting

Currently, in the time-series forecasting field, one challenging work is improving the performance of long sequence time-series forecasting (LSTF). Predicting long-period gait is more difficult than predicting short-period gait. It can explore the deep-level characteristics of gait data and help people design more optimal control strategies.

Before the Transformer structure was developed, recurrent neural network (RNN) models dominated time-series forecasting [Reference Cao, Li and Li12, Reference Sagheer and Kotb13, Reference Sagheer and Kotb14]. Hochreiter et al. propose a LSTM model for solving the gradient vanishing issues [Reference Hochreiter and Schmidhuber15]. Although the LSTM model is excellent, the low computational efficiency brought about by its cyclic structure and the difficulty of feature extraction when dealing with long sequences limit its application.

Since the Transformer model was proposed in the field of natural language processing [Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin16], its ability to process sequence data has been applied to predict time series [Reference Zerveas, Jayaraman, Patel, Bhamidipaty and Eickhoff17, Reference Xu, Dai, Liu, Gao, Lin, Qi and Xiong18]. Although the Transformer model has brought many breakthroughs, it is expensive to use the self-attention mechanism directly in LSTF because of its L-quadratic memory and computation consumption on L-length inputs/outputs. Many improvements are proposed to solve this problem [Reference Kitaev, Kaiser and Levskaya19, Reference Wang, Li, Khabsa, Fang and Ma20]. Li et al. present LogSparse Transformer which the memory cost of the self-attention mechanism is only $O\left ({L*\left ({log(L)} \right )^{2}} \right )$ [Reference Li, Jin, Xuan, Zhou, Chen, Wang and Yan21]. Beltagy et al. introduce an attention mechanism called Longformer that scales linearly with sequence length, reducing its complexity to $O\left ({L*log(L)} \right )$ [Reference Beltagy, Peters and Cohan22]. A representative example of all relevant improvements is the Informer model proposed by Zhou et al., which predicts the output more quickly and accurately with less memory and computation consumption than previous methods [Reference Zhou, Zhang, Peng, Zhang, Li, Xiong and Zhang23]. The Informer model has one essential breakthrough: the ProbSparse self-attention mechanism.

The above time-series forecasting models significantly reduced the model size and improved forecasting accuracy. However, none is interpretable. Prediction accuracy also needs to be improved.

2.3. Model interpretability

The current methods of interpreting neural networks are mainly divided into gradient-based methods and methods based on adjusting input values [Reference Chen, Song, Wainwright and Jordan24]. The attention mechanism of a neural network multiplies the input with nonzero weights that sum to 1. The attention weight of a variable perceptually represents the importance of the variable. Research on the interpretation of neural networks proves that the value of the attention of the Transformer structure is indeed positively correlated with the importance of variables [Reference Jain and Wallace25, Reference Vig and Belinkov26, Reference Vashishth, Upadhyay, Tomar and Faruqui27]. Many interpretable neural network models for time-series forecasting have been proposed. Oreshkin et al. proposed an interpretable time-series forecasting model named N-BEATS, but it can only predict the time series of a dozen time frames in the future, which cannot meet the needs of long-term series forecasting [Reference Oreshkin, Carpov, Chapados and Bengio28]. Lim et al. proposed Temporal Fusion Transformers, a time-series prediction model that combines LSTM and Transformer structures. The LSTM framework it uses limits its application to predicting long-term series [Reference Lim, Arık, Loeff and Pfister29]. In summary, studying interpretable neural network models for long-term series forecasting is meaningful.

3.1. Attention mechanisms

Ashish et al. propose the Transformer based solely on attention mechanisms. The Transformer uses stacked self-attention and point-wise, fully connected layers for both the encoder and decoder to form an encoder–decoder structure.

The core of the attention mechanism is the scaled dot-product attention. Calculating the scaled dot-product attention requires three parameters: queries $(Q)$ , keys $(K)$ , and values $(V)$ , where $Q \in R^{L_{Q}*d}$ , $K \in R^{L_{K}*d}$ , $V \in R^{L_{V}*d}$ , and d is the input dimension. The formula is Eq. (1):

(1) \begin{equation} Attention\left ({Q,K,V} \right ) = softmax\left ( \frac{Q*K^{T}}{\sqrt{d}} \right )*V \end{equation}

The Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers and outperforms them for translation tasks. The attention mechanism effectively splits the input space, emphasizing only elements relevant to the task. Transformer structure is widely used in text-based and chart-based tasks because of its robust feature extraction ability to sequences.

3.2. Informer

Dot-product attention mechanisms require calculating each dot-product pair, which makes the computational cost of the attention mechanism increase squarely with the linear increase of the sequence length $\left ( L_{Q}, L_{K} \right )$ . This property makes long-term series forecasting expensive in computation. To solve this problem, Haoyi et al. propose ProbSparse Self-attention and apply it to the Informer model. They propose the Query Sparsity Measurement to find the most critical vectors of queries $(Q)$ . The formula is Eq. (2), where $L_{K}$ represents the length of $K$ , $d$ represents the model dimension, and $q_i$ , $k_i$ , and $v_i$ are the $i$ th row in $Q$ , $K$ , and $V$ , respectively:

(2) \begin{equation} M\left ({q_{i},K} \right ) = ln{\sum \limits _{j = 1}^{L_{K}}e^{\frac{q_{i}*{k_{j}}^{T}}{\sqrt{d}}}} - \frac{1}{L_{K}}{\sum \limits _{j = 1}^{L_{K}}\frac{q_{i}*{k_{j}}^{T}}{\sqrt{d}}} \end{equation}

Query Sparsity Measurement is Kullback–Leibler divergence between $q_{i}$ and $K$ . The larger the Query Sparsity Measurement is, the more different $q_{i}$ and $K$ are, and the more critical $q_{i}$ is. Based on the Query Sparsity Measurement, the ProbSparse Self-attention allows each $K$ only to attend to the $u$ -dominant $q_{i}$ . The formula is Eq. (3):

(3) \begin{equation} Attention\left ({Q,K,V} \right ) = softmax\left ( \frac{\overline{Q}*K^{T}}{\sqrt{d}} \right )*V \end{equation}

$\overline{Q}$ is a sparse matrix only containing the top- $u$ vectors of $Q$ measured by the Query Sparsity Measurement. $u\,=\,c\,\cdot \,ln\,L_{Q}$ where $c$ is a constant sampling factor. Then, the ProbSparse Self-attention only calculates $O\left ( ln\,L_{Q} \right )$ dot-product for each query-key pair, and the layer memory usage is reduced to $O\left ( L_{K}\,ln\,L_{Q} \right )$ . For self-attention $ L_{Q} = L_{K} = L$ , so the complexity and space complexity of the ProbSparse self-attention is $O(L\,ln\,L)$ .

Informer adopts the self-attention distilling mechanism, adding a convolutional layer on the time dimension between the attention modules. A max-pooling layer with two strides is added to down-sample inputs into its half slice. The distilling mechanism reduces memory usage to $O\left ( (2\, - \epsilon )L\,log\,L \right )$ while ensuring computational accuracy.

3.3. Distilling layer

The distilling layer is a 1D convolutional layer whose output sequence’s length is half the length of the input sequence. The distilling layer reduces the sequence length, memory consumption, and computational complexity. It is also possible to further reduce the sequence length to a quarter or less, which needs to be determined according to the prediction accuracy and scale of the model. Finding the sequence segment with the most information is crucial for interpreting the gait trajectory prediction model. In this paper, we focus on methods to measure the importance of the temporal dimension. So we choose a one-dimensional convolutional network instead of a two-dimensional convolutional network:

(4) \begin{equation}{C_{dis}}_{\frac{L}{2} \times d} ={Conv1d}_{C}\left ( C_{L \times d} \right )\,\,\,\,\,\,\,\,C_{L \times d} \in \left \{{Q,K,V} \right \} \end{equation}

The distilling layer in the IC-former model has three important functions:

1. 1. The distilling layers reduce the length of the input to half, significantly reducing the model size. Reducing model size is especially important when the input and predicted sequences are long.

2. 2. The distilling layers effectively extract context-related features, which helps to improve the prediction accuracy of the model.

3. 3. The distilling layers retain the mathematical meaning of the input. The convolutional network’s local calculation characteristics make the model’s calculation process easier for humans to understand.

3.4. Concatenation operation

The Concatenation operation is an operation that splices two pieces of data together in the time dimension. It is worth noting that there are Concatenation operations inside and outside the attention layer of the IC-former model. Concatenation operations play different roles inside and outside the attention layer:

(5) \begin{equation} A_{cat} = \mathbf{C}\mathbf{o}\mathbf{n}\mathbf{c}\mathbf{a}\mathbf{t}\mathbf{e}\mathbf{n}\mathbf{a}\mathbf{t}\mathbf{i}\mathbf{o}\mathbf{n}\left \{ \left ({Input_A,Input_B} \right )\!,\mathbf{d}\mathbf{i}\mathbf{m} = \mathbf{t}\mathbf{i}\mathbf{m}\mathbf{e} \right \} \end{equation}

The function of the Concatenation operation inside the attention layer is to splice the process variable queries (Q) with the original attention. It enables queries (Q) to continue participating in subsequent calculations as primary features instead of being discarded directly, breaking the original barriers between data. Concatenation operation outside the attention layer splices the auxiliary and main channels in the time dimension to obtain the concatenated information. The attention calculated for the concatenated information represents the relative importance of each input segment.

3.5. Interpretable Multi-head Attention

Although the traditional Transformer structure has proved its powerful feature extraction ability, it destroys the mathematical meaning of the data. The Transformer model cannot measure the data’s importance because of the following two reasons: 1: The queries $(Q)$ , keys $(K)$ , and values $(V)$ in the traditional Transformer structure are obtained by the fully connected layer. But the fully connected layer directly destroys its mathematical meaning. 2: The traditional Transformer structure adds the output of dot-product attention with its original input at the element level, which keeps the original data features. However, it also causes the original data features to continue participating in the subsequent calculation process without being adjusted by the attention weight. The addition at the element level also confuses the mathematical meaning of the data.

In response to these two points, we made adjustments and proposed Interpretable Multi-head Attention. There are two main improvements of Interpretable Multi-head Attention:

1. 1. Get queries $(Q)$ , keys $(K)$ , and values $(V)$ through the distilling layer, which has three advantages. The characteristics of the local calculation of the convolutional network ensure that the mathematical meaning of the obtained queries $(Q)$ , keys $(K)$ , and values $(V)$ is clear and can be intuitively understood by humans. The convolutional network is suitable and efficient for processing sequence information. The data length of queries $(Q)$ , keys $(K)$ , and values $(V)$ obtained by the distilling layer is half of the input, reducing the model’s memory usage.

2. 2. Interpretable Multi-head Attention no longer adds the original input to the output of the attention mechanism at the element level, which ensures that all the data features of the model are obtained based on the weight-adjusted data.

Each column of the attention weight matrix represents the importance of the data corresponding to each row of values $(V)$ . The weight matrix of the attention mechanism calculated by queries $(Q)$ and keys $(K)$ is represented by W, and the elements of each matrix are represented by lowercase letters. L is the length of the input sequence, and d is the model dimension:

(6) \begin{equation} W\, = \,Q\,*\,K = \left[\begin{array}{c@{\quad}c@{\quad}c} w_{1,1} & \cdots & w_{1,L} \\[5pt] \vdots & \ddots & \vdots \\[5pt] w_{L,1} & \cdots & w_{L,L} \end{array}\right] \end{equation}

(7) \begin{align} A &= \,W\,*\,V = \left[\begin{array}{c@{\quad}c@{\quad}c} w_{1,1} & \cdots & w_{1,L} \\[5pt] \vdots & \ddots & \vdots \\[5pt] w_{L,1} & \cdots & w_{L,L} \end{array}\right]\ast \left[\begin{array}{c@{\quad}c@{\quad}c} v_{1,1} & \cdots & v_{1,d} \\[5pt] \vdots & \ddots & \vdots \\[5pt] v_{L,1} & \cdots & v_{L,d} \end{array}\right]\nonumber \\[5pt] &= \left[\begin{array}{c@{\quad}c@{\quad}c} {w_{1,1}*v_{1,1} + \ldots + w_{1,L}*v_{L,1}} & \cdots &{w_{1,1}*v_{1,d} + \ldots + w_{1,L}*v_{L,d}} \\[5pt] \vdots & \ddots & \vdots \\[5pt] {w_{L,1}*v_{1,1} + \ldots + w_{L,L}*v_{L,1}} & \cdots & w_{L,1}*v_{1,d} + \ldots + w_{L,L}*v_{L,d} \end{array}\right] \end{align}

The multi-head attention mechanism divides the data’s model dimension into multiple modules, which does not destroy the mathematical meaning of the 1D convolutional layer. The attention of the whole model dimension is the summing of all multi-head attention. Let H denote the total number of heads, and h represents the hth head:

(8) \begin{equation} W^{h}\, = \,Q^{h}\,*\,K^{h} = \left[\begin{array}{c@{\quad}c@{\quad}c} {w_{1,1}}^{h} & \cdots &{w_{1,L}}^{h} \\[5pt] \vdots & \ddots & \vdots \\[5pt] {w_{L,1}}^{h} & \cdots &{w_{L,L}}^{h} \end{array} \right]\end{equation}

(9) \begin{equation} \begin{split} A^{h}\, = \,W^{h}\,*\,V^{h}\, = \begin{bmatrix}{w_{1,1}}^{h}\;\;\;\;\; & \cdots\;\;\;\;\; &{w_{1,L}}^{h} \\[5pt] \vdots\;\;\;\;\; & \ddots\;\;\;\;\; & \vdots \\[5pt] {w_{L,1}}^{h}\;\;\;\;\; & \cdots\;\;\;\;\; &{w_{L,L}}^{h} \\[5pt] \end{bmatrix}*\begin{bmatrix}{v_{1,1}}^{h}\;\;\;\;\; & \cdots\;\;\;\;\; &{v_{1,d/H}}^{h} \\[5pt] \vdots\;\;\;\;\; & \ddots\;\;\;\;\; & \vdots \\[5pt] {v_{L,1}}^{h}\;\;\;\;\; & \cdots\;\;\;\;\; &{v_{L,d/H}}^{h} \\[5pt] \end{bmatrix} \\[5pt] = \,\begin{bmatrix}{{w_{1,1}}^{h}*{v_{1,1}}^{h} + \ldots +{w_{1,L}}^{h}*{v_{L,1}}^{h}}\;\;\;\;\; & \cdots\;\;\;\;\; &{{w_{1,1}}^{h}*{v_{1,d/H}}^{h} + \ldots +{w_{1,L}}^{h}*{v_{L,d/H}}^{h}} \\[5pt] \vdots\;\;\;\;\; & \ddots\;\;\;\;\; & \vdots \\[5pt] {{w_{L,1}}^{h}*{v_{1,1}}^{h} + \ldots +{w_{L,L}}^{h}*{v_{L,1}}^{h}}\;\;\;\;\; & \cdots\;\;\;\;\; &{{w_{L,1}}^{h}*{v_{1,d/H}}^{h} + \ldots +{w_{L,L}}^{h}*{v_{L,d/H}}^{h}} \\[5pt] \end{bmatrix} \end{split} \end{equation}

(10) \begin{equation} W\, = \,{\sum \limits _{h = 1}^{h = H}W^{h}} \end{equation}

To further reduce the computational complexity and memory consumption, we use the ProbSparse attention proposed in [Reference Zhou, Zhang, Peng, Zhang, Li, Xiong and Zhang23] to replace the traditional dot-product attention.

3.6. Cross Multi-head Attention

The basic structure of Cross Multi-head Attention is similar to that of Interpretable Multi-head Attention. They both use the distilling layer and Concatenation operations. There are only two differences between them. In the Cross Multi-Head Attention, keys $ (k)$ and values $ (v)$ are calculated by the encoder feature and queries $ (q)$ are calculated by the decoder feature. Cross Multi-head Attention retains the addition of the input and output of the attention mechanism at the element level. Cross Multi-head Attention effectively fuses the features of the encoder and decoder.

3.7. Encoder and decoder

The IC-former model is of an encoder–decoder structure. The encoder and decoder are obtained by repeatedly stacking the basic modules mentioned above according to the relationship shown in Fig. 3. The number of stacked modules can be flexibly adjusted based on data volume and model size.

We denote the length of the input sequence by L. The larger the value of L, the greater the consumption. This shortcoming is especially serious for dot-product attention because of L-quadratic memory and computation consumption on L-length inputs/outputs. To alleviate this problem, we use distilling layers in the attention layer and the encoder and decoder further to reduce the memory and computation consumption of the model.

The encoder and the decoder have two channels: main and auxiliary channels. The main channel measures the importance of the input sequence. The auxiliary channel provides the input with a precise mathematical meaning. Because the auxiliary channel only contains distilling layers, the data it processes retain a clear mathematical meaning. A Concatenation operation between the encoder and decoder concatenates the uninterpretable main channel and the interpretable auxiliary channel to obtain spliced features. The spliced features are input to the next module’s Interpretable Multi-head Attention module, and each fragment’s importance is measured. The attention of the interpretable auxiliary channel represents the relative richness of information in each sequence segment. The importance of the interpretable auxiliary channel features is also global, indicating the relative information abundance of each sequence segment in the model.

After the sequence passes through the stacked distilling layers in the auxiliary channel, the sequence length is shortened layer by layer. The granularity of the sequence continues to increase, which means that the length of the original input sequence represented by each vector increases accordingly. IC-former compares the relative importance of each fragment at different granularities. Measuring the local and global importance of sequence fragments reveals the data basis of the neural network model.

The structure of the decoder and the encoder is similar. The main difference is that the decoder has a Cross Multi-head Attention module, which fuses the features of the encoder and decoder. The input to the encoder is the entire input sequence. The input to the decoder is a combined sequence obtained by concatenating the input sequence and a zero sequence with the same length as the output sequence. The final decoder feature is input to a fully connected layer to obtain the final prediction output.

4.1. Accuracy test

4.1.2. Experimental details

Baselines: Zhou et al. tested Informer [Reference Zhou, Zhang, Peng, Zhang, Li, Xiong and Zhang30], ARIMA [Reference Ariyo, Adewumi and Ayo31], Prophet [Reference Taylor and Letham32], LSTMa [Reference Bahdanau, Cho and Bengio33], LSTnet [Reference Lai, Chang, Yang and Liu34], and DeepAR [Reference Salinas, Flunkert, Gasthaus and Januschowski35] on the above-mentioned datasets. We cite Zhou et al.’s experimental results to compare with those obtained by our proposed IC-former on the above datasets.

Although our proposed model can effectively multivariate and univariate forecasting, we focus on univariate forecasting. We believe predicting gait trajectory should depend on as few sensors as possible. Fewer sensors mean fewer hardware requirements, which can significantly reduce the application threshold of algorithms. In the part of experiments, the IC-former model contains a three-layer encoder and a two-layer decoder. In the other part of the experiments, the IC-former model includes a two-layer encoder and a one-layer decoder. The IC-former model is optimized with Adam optimizer, and its learning rate starts from 0.0001, decaying two times smaller every two epochs, and the total epochs are 20. We set the batch size to 32.

We used two evaluation metrics, $MSE$ and $MAE$ , on each prediction window (averaging for multivariate prediction) and rolled the whole set with $stride = 1$ :

\begin{equation*} MSE=\frac {1}{n}\sum _{i=1}^{n}\left (y-\hat {y}\right )^2 \end{equation*}

\begin{equation*} MAE=\frac {1}{n}\sum _{i=1}^{n}\left |y-\hat {y}\right | \end{equation*}

The prediction window sizes in ETTH, ECL, and Weather are $\left \{1d,2d,7d,14d,30d,40d\right \}$ . The prediction window size in ETTm is $\left \{6h,12h,24h,72h,168h\right \}$ . Our computing platform (Nvidia 3090 24 GB GPU) is a little worse than the platform (Nvidia V100 32 GB GPU) used in the paper [Reference Zhou, Zhang, Peng, Zhang, Li, Xiong and Zhang30], but we still completed all prediction tasks, which shows that our model is adequate.

4.2. Gait trajectory prediction