A) Group delay function

The GD is defined as the negative derivative of the unwrapped phase spectrum [Reference Murthy and Gadde25]:

(1)\begin{equation} \tau_l(\omega) = -\frac{d(\theta_l(\omega))}{d\omega}, \end{equation}

where $\omega$ is the frequency variable, $\theta _l(\omega )$ is the phase spectrum of the signal $x_l(n)$, which is the $l$-th frame of the entire signal $x(n)$. However, it is difficult to estimate the phase spectrum due to the phase wrapping problem. Thus, the GD is usually calculated without the phase spectrum, as follows [Reference Murthy and Gadde25]:

(2)\begin{equation} \begin{aligned} \tau_l(\omega) & = -\frac{ d(\theta_l(\omega)) } { d\omega } \\ & = - Im\left( \frac{ d(log(|X_l(\omega)|) + j\theta_l(\omega)) } { d\omega } \right) \\ & = - Im\left( \frac{ d(log(X_l(\omega))) } { d\omega } \right) \\ & = Re\left( \frac{ Y_l(\omega) } { X_l(\omega) } \right) \\ & = \frac{ Re(Y_l(\omega))Re(X_l(\omega)) + Im(Y_l(\omega))Im(X_l(\omega)) } { |X_l(\omega)|^{2} }, \end{aligned} \end{equation}

where $Re(\bullet )$ and $Im(\bullet )$ are the real and imaginary part of a complex variable, respectively. $X_l(\omega )$ is the spectrum of the signal $x_l(n)$ and $Y_l(\omega )$ is the first derivative of $X_l(\omega )$, which can also be seen as the spectrum of the signal $nx_l(n)$.

B) Modified GD

When the energy of a signal (the $|X_l(\omega )|^{2}$ in equation (2)) is close to zero, there will be spiky peaks in the GD. MGD can solve this problem by using the cepstrally smoothed spectrum, called $S_l(\omega )$, as the denominator of $\tau _l(\omega )$ [Reference Murthy and Gadde25]:

(3)\begin{align} \hat \tau_l(\omega) = sign. \left| \frac{ Re(Y_l(\omega))Re(X_l(\omega)) + Im(Y_l(\omega))Im(X_l(\omega)) } { |S_l(\omega)|^{2\gamma} } \right| ^{\alpha}, \end{align}

where $sign$ is the sign of the original GD, $\gamma$ and $\alpha$ are two tunable parameters, varying from 0 to 1.

C) CQT-based MGD feature

The traditional MGD function is extracted with the Fourier transform. The Fourier transform is related to the CQT. Both of them can be seen as a bank of sub-band filters. The main differences between CQT and the Fourier transform is that (1) the center frequencies of CQT filters are geometrically spaced as $f_k = 2^{{k}/{b}} f_0$, where $f_0$ is the minimal frequency, and $b$ is the number of filters per octave, and (2) the bandwidth of each CQT filter is determined by the center frequency as $\delta _k = f_{k+1} - f_k$, resulting a constant ratio of center frequency to resolution $Q=({f_k})/({\delta _k})=(2^{{1}/{b}}-1)^{-1}$. The CQT is defined as:

(4)\begin{equation} X^{cqt}(k) = \frac{ 1 } { N_k } \sum_{n=0}^{N_k-1} x(n) W_{N_k}(n) e^{-j 2 \pi Q n/ N_k}, \end{equation}

where $N_k = Q ({f_s})/({f_k})$ is the length of analysis windows, $f_s$ is the sample rate of the signal, $W_{N_k}(\bullet )$ is an arbitrary $N_k$-length window function. As a comparison, the definition of the discrete-time Fourier transform is:

(5)\begin{equation} X^{ft}(\omega) = \frac{1} {N} \sum_{n=0}^{N-1} x(n) W_{N}(n) e^{-j \omega n}. \end{equation}

It will be found that $X^{cqt}(k) = X^{ft}(({2\pi Q})/({N_k}))$ when $N=N_k$. That means CQT can be seen as a type of the Fourier transform with carefully designed varying-length windows and a constant Q-factor, making it reasonable to extract MGD with CQT.

Realizing the relationship between CQT and the Fourier transform, CQT-based MGD can be seen as a bunch of MGD with variable-length frames, i.e.

(6)\begin{equation} \hat\tau_l^{cqt}(k) = \hat\tau_{l,N_k}\left(\frac{2 \pi Q}{N_k}\right), \end{equation}

where $\hat \tau _{l, N_k}(\bullet )$ is the MGD function at the $l$-th $N_k$-length frame. However, frequency-by-frequency frame-by-frame calculations are time-consuming. To utilize the fast algorithm of CQT [Reference Brown and Puckette26] to accelerate the extraction process, an auxiliary signal $y'(n)$ is introduced as

(7)\begin{equation} y'(n) = nx(n). \end{equation}

For each frame of $y'(n)$, we have

(8)\begin{equation} y_l'(n) = (l \cdot T + n)x_l(n), \end{equation}

where T is the hop size between two adjust frames. Thus, we have

(9)\begin{equation} Y_l(\omega) = Y_l'(\omega) - l \cdot T \cdot X_l(\omega). \end{equation}

Since both $Y'(\omega )$ and $X(\omega )$ can be calculated by the fast algorithm of CQT, CQT-based MGD can be extracted according to equation (3), as shown in Algorithm 1.

Algorithm 1 CQTMGD ExtractionMethod

A) Residual neural network

Residual neural network (ResNet) is widely used in image recognition fields and also explored in the replay detection task [Reference Chen, Xie, Zhang and Xu12]. It eases the optimizing of a deep network by using residual mapping instead of direct mapping. Formally, it tries to learn a residual mapping function $F(x)$ instead of learning the desired underlying mapping $H(x)$ directly, where $H(x) = F(x)+x$. This overcomes the performance degradation when plain CNNs going deeper.

ResNet is constructed by stacking building blocks. In [Reference He, Zhang, Ren and Sun27], two different kinds of building blocks are proposed for ResNet, that is, the basic building block and bottleneck block. The basic block, which consists of two convolutional layers, is used in shallow networks, such as ResNet-18 and ResNet-34. The bottleneck block, which consists of three convolutional layers, is used in deep networks, such as ResNet-50.

B) Multi-branch ResNet

In this paper, a multi-branch building block is proposed. The input of the building block will be processed by K independent paths. The output of each path is concatenated to form the final output of the building block. Due to the independence of each path, the depth of each convolutional kernel is decreased. Thus, the model can be compressed by the multi-branch structure. It is important for replay detection because the limited training data may be overfitted easily. The multi-branch building block is defined as:

(10)\begin{equation} \textbf{Y} = \textbf{X} + \mathop{\Xi}_{i=1}^{D} f_i(\textbf{X}), \end{equation}

where $\textbf {Y}$ is the output of the building block, $\textbf {X}$ is the input tensor, $f_i(\bullet )$ can be an arbitrary function which splits the input first and then transforms them, $D$ is the number of transformations, $\mathop {\Xi }$ is an aggregate function which concatenates a tensor along the channel-axis, defined as:

(11)\begin{equation} \begingroup \mathop{\Xi}_{i=1}^{D}[S^{(i)}_1, \ldots, S^{(i)}_{C_i}] = [S^{(1)}_1, \ldots, S^{(1)}_{C_1}, S^{(2)}_1, \ldots, S^{(D)}_{C_D}], \endgroup \end{equation}

where $[S^{(i)}_1,\ldots ,S^{(i)}_k]$ is a k-channel tensor.

A residual neural network with the multi-branch building block proposed above is named ResNeWt. The “W” in the name is the concatenation of the two characters “v”. It implies the multi-branch structure which processes input values separately and concatenates the processing results as the output. The proposed ResNeWt-18, which is based on ResNet-18, is shown in Table 1. There are three main differences: (1) the multi-branch structure is used based on the basic block, (2) the number of filters in the building block is doubled, and (3) a dropout layer is added after global average pooling layer. The spectral feature of each utterance is repeated or/and truncated along the time-axis to ensure the shape of inputs is fixed. Then it is fed into the network to determine whether the utterance is genuine or not.

Table 1. The overall architecture of ResNeWt18. The shape of a residual block [Reference He, Zhang, Ren and Sun27] is inside the brackets, and the number of stacked blocks on a stage is outside the brackets. “C=32” means the grouped convolutions [Reference Krizhevsky, Sutskever and Hinton28] with 32 groups. “2-d fc” means a fully connected layer with 2 units.

There are two commonly used multi-branch neural networks, that is, the Inception network [Reference Szegedy, Ioffe, Vanhoucke and Alemi29] and ResNeXt [Reference Xie, Girshick, Dollár, Tu and He30]. The main difference between the proposed multi-branch structure and the Inception network is the structure of branches. In ResNeWt, all the branches follow the same topological structure. While in Inception, they are different. We use the same structure because it eases the hyper-parameter tuning progress. Comparing ResNeWt with the ResNeXt, both follow the split-transform-merge strategy to build a multi-branch structure. The difference is the aggregate function. In the ResNeXt, it uses the sum function. In ResNeWt, it uses the concatenate function. Thus, the building blocks of ResNeXt need to have more than three layers. Otherwise, it should equal to a wide and dense module [Reference Xie, Girshick, Dollár, Tu and He30]. In contrast, our proposed structure can be applied to an arbitrary number of layers. Thus, the two-layer basic block of ResNet can only be used in ResNeWt.

A) Database

The ASVspoof 2019 physical access dataset was used for evaluating the proposed system in this paper. Table 2 describes the subset configuration. This dataset is based upon simulated and carefully controlled acoustic and replay configurations for the convenience of analysis. The simulation processes are illustrated in Fig. 1. The VCTK corpusFootnote ¹ was used as the source speech data. Bona fide attempts were simulated by passing the source speeches to an environment simulator. Replay spoofing attempts were simulated by passing the source speeches to an environment simulator first (to simulate recording environments), followed by a device simulator (to simulate playback devices), and then another environment simulator (to simulate playback environments). The environment was assumed to be a room simulated by RoomsimoveFootnote ² with three parameters, that is, the room size $S$, reverberation time $RT$, and the distance from the talker to the microphone $D$. The effects of devices were represented by a quality indicator $Q$, which was simulated by the generalized polynomial Hammerstein model and the synchronized sweptsine toolFootnote ³.

Table 2. A summary of the ASVspoof 2019 physical access dataset [31].

Fig. 1. An illustration of the simulation processes in the ASVspoof 2019 physical access dataset (adapted from [31]).

B) System description

This subsection provides the detailed description of the implemented system.

(1) Feature Extraction: Two categories of feature sets, namely, magnitude-based and phase-based features, were extracted using STFT and CQT. For magnitude-based features, we extracted the STFT-based log power magnitude spectrogram (Spectrogram), Mel scale filter banks (MelFbanks), and CQT-based log power magnitude spectrogram (CQTgram). For phase-based features, we extracted the MGD feature and the proposed CQT-based MGD (CQTMGD) feature. Spectrogram and MelFbanks were extracted with Hamming window, 50 ms frame length, 32 ms frameshift, and 1024 FFT points. A total of 128 Mel filter banks were extracted in MelFbanks. The MGD feature was extracted with 50 ms frame length, 25 ms frameshift, Hamming window, 1024 FFT points, $\alpha =0.6$, and $\gamma =0.3$. The CQTgram and CQTMGD features were extracted with 32 ms frameshift, Hanning window, 11 octaves, and 48 bins per octave. For CQTMGD, we set $\alpha =0.35$ and $\gamma =0.3$. All the features were truncated along the time-axis to preserve exactly 256 frames. The short speeches were extended by repeating itself. Finally, all the inputs were resized to $512\times 256$ by the bilinear interpolation.

2) Training Setup: ResNeWt was optimized by the Adam algorithm with $10^{-3.75}$ as the learning rate, and the batch size was 16. The training process was stopped after 50 epochs. The loss function was the binary cross-entropy between predictions and targets. The dropout ratio was set to 0.5. The output of the “bonafide” node in the last full connection layer was obtained as the decision score (before the softmax function). The Pytorch toolkit [Reference Paszke, Gross, Chintala, Chanan, Yang, DeVito, Lin, Desmaison, Antiga and Lerer32] was employed to implement the model.

3) Score Fusion: A score-level fusion was performed to combine the models using different features. For simplicity, the ensemble system averaged the output score of all the subsystems. A greedy-based strategy was used in selecting subsystems. First, the best system was chosen. Then, each time, one system was added and the new ensemble system was evaluated in the development set. The best one was selected according to the minimum normalized tandem detection cost function (min-tDCF) (described in Section V.C). The selection process would not stop until the performance no longer improved.

C) Performance evaluation

In the ASVspoof 2019 challenge, the minimum normalized tandem detection cost function (min-tDCF) [31, Reference Kinnunen, Lee, Delgado, Evans, Todisco, Sahidullah, Yamagishi and Reynolds33] was used as the primary metric, which can be calculated as:

(12)\begin{equation} \text{t-DCF}^{\text{min}}_{\text{norm}} = \min\limits_{s} \beta P_\text{miss}^{\text{cm}}(s) + P_\text{fa}^{\text{cm}}(s) , \end{equation}

where $P_\text {miss}^{\text {cm}}(s)$ and $P_\text {fa}^{\text {cm}}(s)$ are, respectively, the miss rate and false alarm rate of a countermeasure (CM) system at threshold $s$; $\beta$ is a cost which depends on the min-tDCF parameters and ASV errors ($\beta \approx 2.0514$ in the ASVspoof 2019 physical access development set with the ASV scores provided by the organizers of the challenge [31]).

The equal error rate (EER) [31] was also used as the secondary metric.

A) Comparison of different features

Table 3 depicts a quantitative comparison of different features. Among all the magnitude-based features, CQTgram achieved the lowest EER and almost the lowest min-tDCF. The concatenation of CQTgram and MelFbanks improved the min-tDCF slightly, increasing the EER in the meantime. However, as shown in Fig. 2, the output of CQTgram-based (B) and concatenated feature-based (C) systems were highly correlated. Thus, CQTgram may be the key contributor in the concatenated feature. For the phase-based features, CQTMGD outperformed MGD, which should be attributed to CQT. It indicates that CQT outperformed the Fourier transform in this dataset/task. Finally, score-level fusion achieved further improvements, indicating the complementarity between magnitude and phase, as well as between CQT and the Fourier transform.

Table 3. Performance in the ASVspoof 2019 physical access development set with different features using ResNeWt18 as the classifier.

^aA|B: Concatenating the feature set A and B along the frequency-axis with the shape of $656 \times 256 (656 = 528\hbox{(CQTgram)} + 128\hbox{(MelFBanks)})$.

^bC+D+E: The fusion (score averaging) of subsystems.

Fig. 2. The correlation between the decision score of the systems using different features in the ASVspoof 2019 physical access development set. A|B is the concatenating of feature A and B along the frequency-axis.

B) Comparison with related models

Table 4 shows the performance achieved by different classifiers in the development set. ResNeWt18 outperformed ResNet18 on the three types of features. ResNeWt34 outperformed ResNet34 on two types of features (CQTgram and CQTGMD). The similar results were achieved for ResNeWt50 and ResNet50. Those results indicate the effectiveness of ResNeWt. However, when comparing ResNeWt50 with ResNeXt50, we found ResNeXt50 achieved better performance. Since the multi-branch structure of ResNeXt cannot be applied to two-layer building blocks, ResNeWt is a good supplement in this field. By comparing the models which use the different numbers of layers, it can be found that the shallow model worked better than the deep model. Based on the results, we chose to use the ResNeWt18 model as the classifier in the rest of the paper.

Table 4. Performance (EER%) in the ASVspoof 2019 physical access development set with different models.

C) Comparison with relevant systems

Table 5 compares the performance of our proposal to other relevant systems. Our proposed system outperformed the baseline systems, as well as other top-performing systems. In particular, the system that used the concatenating of CQTgram and MelFbanks feature as inputs already outperformed other top-performing fusion systems. Compared with the best baseline system, our best fusion system yielded 96.1% and 96.5% relative error reduction on min-tDCF and EER, respectively. Also, it achieved 21.3% and 27.8% relative error reduction on min-tDCF and EER, respectively, compared with the fusion system proposed in [Reference Lavrentyeva, Novoselov, Tseren, Volkova, Gorlanov and Kozlov34].

Table 5. Comparison with relevant systems in the ASVspoof 2019 physical access evaluation set.

D) Performance in ASVspoof 2017 V2

To explore the generalization of the proposed system, we further evaluated it in the ASVspoof 2017 V2 dataset. Due to the limited amount of the training data in the ASVspoof 2017 V2 dataset, the training progress was stopped after 25 epochs. Both the training and development set in the ASVspoof 2017 V2 dataset was used for training.

The experimental results are shown in Table 6. Firstly, both MGD and CQTMGD performed worse than the baseline systems, but the fusion of them outperformed the best baseline system. It shows that both feature sets are informative but not informative enough for replay detection. Secondly, the best single system outperformed the baseline systems, which shows the generalization of the proposed method. Thirdly, the fusion of the four feature sets further improved the performance, which shows the complementarity between those feature sets. Fourthly, overall performance was much worse than the performance reported in the ASVspoof 2019 dataset, which indicates that the real-scene replay data are more challenging than the simulated replay data. Lastly, the Spectrogram feature set outperformed the CQTgram feature set, and MGD outperformed CQTMGD. Those observations may indicate that the Fourier transform outperforms CQT in this dataset. This finding goes against the findings in the ASVspoof 2019 dataset (in Section VI.A). We will analyze it in the discussion section.

Table 6. Comparison with relevant systems in the ASVspoof 2017 V2 evaluation set.