A) Original OM-LSA

The MMSE-STSA speech estimator was based on the idea that the amplitude of each frequency component X(k, l) is a random variable, and the estimate ${\hat{X}}(k,l)$ ) should minimize

(1) $$E\{(\vert X(k,l) \vert - \vert \hat{X}(k,l) \vert)^2\}$$

where (k, l) is the frequency component and frame indices.

The log spectral amplitude (LSA) estimator is the improved version of MMSE-STSA, in which the cost function is defined by the difference of log X:

(2) $$E\{(\log \vert X(k,l) \vert -\log \vert \hat{X}(k,l)\vert)^2\},$$

which is known to be more suitable for speech processing.

Solving (2) is mathematically complicated but straightforward, and the solution is expressed as follows.

(3) $$\eqalign{ \vert \hat{X}(k,l)\vert &= G_H(k,l)\vert Y(k,l)\vert,}$$

(4) $$\eqalign{G_H(k,l) &= \displaystyle{{\xi(k,l)}\over{1+\xi(k,l)}}\exp\left(\displaystyle{{1}\over{2}}\int_{\nu(k,l)}^{\infty}\displaystyle{{{\rm e}^{-t}}\over{t}}dt\right),}$$

(5) $$\eqalign{\nu(k,l) &= \gamma(k,l)\xi(k,l)/(1+\xi(k,l)),}$$

where Y(k, l) is the amplitude of observed signal in the frequency domain, and G _H (k, l) is the gain function. Variables ξ(k, l) and γ(k, l) are called a priori SNR and a posteriori SNR, respectively. The a posteriori SNR is defined by the current frame observation;

(6) $$\gamma(k,l) = \displaystyle{{\vert Y(k,l) \vert^2}\over{\sigma_m^2(k,l)}}$$

where the noise process variance $\sigma_{m}^{2}(k,l)$ must be obtained separately. As in [Reference Cohen and Berdugo20], we use the minima-controlled recursive averaging (MCRA) noise estimation to obtain $\sigma_{m}^{2}(k,l)$ . The a priori SNR is estimated recursively using the estimated gain of the previous frame;

(7) $$\eqalign{\xi(k,l)& = c_1G_H^2(k,l-1)\gamma(k,l-1) \cr &\quad + (1-c_1) \max\{\gamma(k,l)-1, 0\}}$$

where c ₁ is an adjustable weight parameter. A more detailed derivation of the above-mentioned solution could be found in [Reference Ephraim and Malah23].

OM-LSA modifies the result of LSA by taking the weighted geometric mean of G _H (k, l) and its lower boundary $G_{{\rm min}}$ , where the weight is based on the speech presence probability p(k, l).

(8) $$\eqalign{\vert \hat{X}(k,l) \vert &= G(k,l)\vert Y(k,l)\vert}$$

(9) $$\eqalign{G(k,l) &= [G_H(k,l)]^{p(k,l)}G_{{\rm min}}^{1-p(k,l)}. }$$

The speech presence probability is obtained by

(10) $$p(k,l) = \left[ {1 + \displaystyle{{q_0} \over {1 - q_0}}(1 + \xi (k,l)){\rm e}^{ - \nu (k,l)}} \right]^{ - 1}$$

where q ₀ is the a priori speech absence probability.

After the noise suppression process of OM-LSA was done, there are two approaches to obtain VAD results. The first approach is to reconstruct the waveform of the noise-suppressed signal by combining the estimated amplitude ${\hat{X}}(k,l)$ and the original phase. Once the reconstructed waveform was obtained, any existing VAD method can be applied. The simplest example is to classify speech and non-speech frames using a power threshold, which we discuss in the next section. It is also possible to apply more sophisticated machine learning-based method, which we discuss in Section IV. The second approach is to use the speech presence probability p(k, l) defined by (10) or the likelihood ratio $\Lambda(k,l)$ used in [Reference Sohn, Kim and Sung17]. They can be integrated to the framewise score as follows.

(11) $$\eqalign{P(l) &= \prod_{k=0}^{K-1} p(k,l),}$$

(12) $$\eqalign{\Lambda(l) &= \prod_{k=0}^{K-1} \Lambda(k.l), \cr & = \prod_{k=0}^{K-1} \displaystyle{{1}\over{1+\xi(k,l)}}\exp \displaystyle{{\gamma(k,l)\xi(k,l)}\over{1+\xi(k,l)}}}$$

where K is the number of frequency components. VAD can be realized simply by applying a threshold for these framewise scores.

B) Augmented implementation of OM-LSA

The original OM-LSA estimator was intended to provide better speech signal for communication and recognition. Therefore, it was designed to maintain the balance between lowering the noise level and not causing the distortion. However, reducing the noise is particularly important for VAD, while the distortion is rarely harmful. In view of the priority of noise removal, we propose to use some augmentation techniques for the OM-LSA speech estimator.

The first step of augmentation focuses on (6). Although MCRA is known to be a reliable noise estimation algorithm, we believe that the noise level must be over-estimated so that any unreliable part of the spectrogram does not affect the VAD results. In fact, we formerly found that the noise level must be under-estimated to obtain more accurate results of speech recognition [Reference Obuchi, Takeda and Togami24], because distortion is the main cause of misrecognition. Both under and over estimation of the noise can be realized by introducing a weight factor α as follows.

(13) $$\gamma(k,l) = \displaystyle{{\vert Y(k,l)\vert^2}\over{\alpha\sigma_m^2(k,l)}}.$$

The second step of augmentation focuses on (8) that defines how the estimated gain is applied to the speech amplitude. Augmentation is realized simply by introducing a modification parameter β as follows.

(14) $$ \vert \hat{X}(k,l)\vert =G^{\beta}(k,l)\vert Y(k,l) \vert.$$

The third step of augmentation is a screening step of the prominent noise component. In contrast that speech has a wideband spectrum, there are kinds of noises that have a very sharp peak in the power spectrum, such as electrical beeps and musical instrument sounds. The prominent component in the power spectrum could be removed by the process described as follows.

(15) $$ \vert \hat X(k,l)\vert = \left\{ {\matrix{ {G^\beta (k,l)\vert Y(k,l)\vert} & {{\rm rank}(k) \ge \eta K} \cr 0 \hfill & {{\rm otherwise},} \hfill \cr } } \right.$$

where rank(k) is the number of frequency components whose magnitude is larger than the kth frequency. At last, our VAD-oriented OM-LSA-based speech estimator is defined by equations (4), (5), (7), (9), (10), (13), and (15).

A) CENSREC-1-C evaluation framework

The proposed VAD algorithm was evaluated using CENSREC-1-C [Reference Kitaoka25], a public VAD evaluation framework. The data part of CENSREC-1-C consists of two datasets: simulated dataset and real dataset. In this paper, we used the real dataset made of Japanese connected digit utterances. The CENSREC-1-C real dataset is further divided into four subsets based on the environment (university restaurant and vicinity of highway street) and SNR level (high and low). Its details are shown in Table 1, where the average noise levels were cited from [Reference Kitaoka25] and the SNR levels were estimated using WADA-SNR [Reference Kim and Stern26].

Table 1. Detail of CENSREC-1-C real dataset.

CENSREC-1-C includes not only the data themselves, but also the voice activity labels and performance calculation tools. When a specific algorithm and corresponding settings are given, the false alarm rate (FAR) and the false rejection rate (FRR) are automatically calculated. FAR is the ratio of incorrectly labeled non-speech frames over the total non-speech frames. FRR is the ratio of incorrectly labeled speech frames over the total speech frames. We also use the average error rate (AER), which is the average of FAR and FRR.

B) Evaluation using classifiers without model training

The first set of experiments using CENSREC-1-C was conducted to compare three framewise scores, P(l), Λ(l), and Q(l). The threshold-based classifier was applied to those three scores obtained by statistical noise suppression without augmentation ( $\alpha=1.0, \beta=1.0, \eta=0.0$ ). Other parameter setting was the same as in [Reference Obuchi, Takeda and Kanda21] ( $C_{1}=0.99, G_{{min}}=0.01, q_{0}=0.2$ ). Figure 1 shows the ROC curves obtained by applying various threshold values. Since CENSREC-1-C provides the baseline VAD tool, the corresponding ROC curve was plotted for comparison. It is clear that all three scores resulted in lower FARs and FRRs than the baseline. Among them, P(l) is clearly less effective, and Q(l) is slightly better than Λ(l). Accordingly, Q(l) is used as the framewise score in this section.

Fig. 1. Comparison of framewise scores. The same threshold-based classifier was applied.

The second set of experiments was to confirm the effectiveness of the augmented statistical noise suppression. The same threshold-based classifier was applied to the output of statistical noise suppression, in which various augmentation was applied.

Figure 2 shows the effectiveness of noise over-estimation, defined by (13). The values of β and η were not augmented, and the value of α was changed. The point α = 1.0 corresponds to the original OM-LSA, and it was reported that $\alpha\approx 0.2$ is optimal for speech recognition [Reference Obuchi, Takeda and Togami24]. For each value of α, various thresholds were tested, ${\rm AER}=({\rm FAR}+{\rm FRR})/2$ was calculated, and the smallest AER was plotted. The results revealed that decreasing the value of α is only harmful for VAD, and AER drops rapidly when α increases from 1.0 to 2.0. When α is further increased, AER seems to be saturated.

Fig. 2. Effect of over-subtraction expressed by various α.

Figure 3 shows the effectiveness of gain nonlinearity, defined by (14). The value of α was fixed at 5.0, and various values of β were tried. In this case, some improvements were observed with increasing β, but it is not as dramatic as in the case of increasing α.

Fig. 3. Effect of gain augmentation expressed by various β.

The results obtained by various values of η are shown in Fig. 4, where β was fixed at 1.4. Again, AER drops rapidly with increasing η, and reached its smallest value at η = 0.07. The best AER was 9.93%, where ${\rm FAR}=12.42\%$ and ${\rm FRR}=7.43\%$ .

Fig. 4. Effect of prominent component removal expressed by various η.

We also investigated the possibility of using adaptive parameters. Obviously, the most effective parameter is the frame power threshold. Since the dataset of CENSREC-1-C is made of four subsets, we checked the AER of our best setting ( $\alpha=5.0, \beta=1.4, \eta=0.07$ ) obtained with the optimal threshold for each subset. In this case, the AERs for Restaurant (SNR high), Restaurant (SNR low), Street (SNR high), and Street (SNR low) were 7.00, 18.03, 2.73, and 3.94% respectively. The average was 7.93%, which is much better than the AER obtained with the fixed threshold (9.93%).

Similar experiments were conducted with the adaptive α, β, and η. In the case of α (see Fig. 2), the lowest AER of 10.44% was still improved to 9.27%. In the case of β (see Fig. 3), the lowest AER of 10.35% was still improved to 9.53%. Finally, in the case of η (see Fig. 4), the lowest AER of 9.93% was improved to 9.37%.

Although these results indicate the potential value of adaptive parameters, their effectiveness are strongly dependent on the correct evaluation of the environment. We must consider the risk of performance degradation caused by the incorrect parameter setting.

C) Classifier training

More accurate VAD results could be achieved by the model-based classifiers. In this paper, we investigate three types of model-based classifiers: DT, SVM, and CNN. We also try k-means clustering to compare supervised and unsupervised training. Before applying them to CENSREC-1-C, we train the classifiers and clusterer using a separate dataset.

1) Training and development data

The dataset for classifier and clusterer training, referred to as Noisy UT-ML-JPN database in this paper, was created using UT-ML public database [27] and our proprietary noise data. The noise data were recorded in a running car and in a cafeteria; they were added to the utterances of Japanese subset of UT-ML database with SNR of 0, 5, and 10 dB. Speech/non-speech labels for Noisy UT-ML-JPN database were generated automatically by applying a power threshold for the clean version of UT-ML database. The labeling process is completely framewise, meaning that inter-frame smoothing was not applied.

The Japanese subset of UT-ML database includes six male and six female speakers, each of which read one long article (54.8 s on average) and 50 short phrases (3.3 s on average). Original data were recorded by 16 kHz sampling rate, but they were downsampled to 8 kHz. One second silences were appended on both ends of utterances before noises were added.

After adding the noise, augmented statistical noise suppression described in Section II-B) was applied. Based on the results of Section V-B) and [Reference Obuchi, Takeda and Kanda21]Footnote ¹ , the parameter setting was α = 5.0, β = 1.4, and η = 0.07. Applying the same noise suppression process to the training and test data are called noise adaptive training (NAT), and known to be effective for speech recognition [Reference Deng, Acero, Plumpe and Huang28]. We also prepare the clean version of the training data for comparison.

Finally, data of one male and one female speakers were used for development, and the other data were used for model training. The size of the training and development set are shown in Table 2.

Table 2. Length and number of frames of Noisy UT-ML-JPN database.

2) Toolkits

The classifiers were trained using publicly available toolkits. WEKA [Reference Hall, Frank, Holmes, Pfahringer, Reutemann and Witten29] was used for DT and SVM training, and Caffe [Reference Jia30] was used for CNN training. The k-means clustering program was prepared by ourselves.

When we use WEKA for DT and SVM, we adopt the classifier ensemble approach. Although we have plenty of training samples (more than 600 k of speech and more than 700 k of non-speech), it takes too long to execute the training using the whole data at once. Therefore, we split the training data into small chunks, and trained many classifiers using different data chunks. The final classification result can be obtained by voting of these small classifiers. CNN training and k-means clustering were executed using all data.

DT by WEKA (J48) runs without any specific parameter adjustment. SVM by WEKA (SMO) requires at least selection of the kernel, so we used the two-dimensional polynominal kernel. CNN by Caffe requires quite a few parameters to be set, but we only show the network topology in Fig. 5, that would be the most important information.

Fig. 5. CNN topology. Relu stands for rectified linear unit. The output layer (ip2) has two units corresponding to speech and non-speech.

3) Preliminary evaluation

Before going back to CENSREC-1-C, we checked the basic performance of three classifiers and one clusterer using the development data. The development data were processed by augmented statistical noise suppression using the same parameters as the training data. The processed data were then divided into frames and classified by either DT, SVM, CNN or k-means.

Figure 6 depicts the results obtained by the classifier ensemble of DT and SVM with various sizes. Inter-frame smoothing was not applied, so the graph shows the pure performance of framewise classification. It should also be noted that the classifier directly classifies the frame data, so we obtain a single pair of FAR and FRR instead of the ROC curve. The results indicated that SVM is superior to DT if only few classifiers were used. However, it was reversed if we have seven or more classifiers. AER was saturated at about 20 classifiers (SVM) or 40 classifiers (DT).

Fig. 6. Preliminary evaluation of classifier ensemble.

Figure 7 shows the comparison of simple thresholding, k-means clustering, and various classifiers. If we apply simple thresholding, AER was 21.29%. In the case of k-means clustering, an additional bias θ was introduced to play the role of adjustable threshold. A frame is labeled as non-speech if $d_{ns}-d_{s}<\theta$ , where d _s (d _ns ) is the distance between the frame and the centroid of speech (non-speech) cluster. Although AER was as high as 27.19% when θ = 0, it becomes 20.44% if the value of θ was optimized. The AERs of DT and SVM were clearly better than simple thresholding and k-means. On the right-hand side of the figure, AERs obtained by various CNNs are plotted, where L is the number of fully-connected intermediate layers (rectangular block including ip1+relu1 of Fig. 5). As we expected, CNN (NAT) provides better results than CNN (clean), indicating that NAT was effective. It is also found that the deeper the network is, the more accurate classification results were obtained.

Fig. 7. Preliminary evaluation by comparing various classifiers.

D) Evaluation using model-based classifiers

Finally, we applied various classifiers to CENSREC-1-C. The evaluation procedure was the same as in the preliminary evaluation, except that inter-frame smoothing was applied. In addition, a gain adjustment factor was multiplied to to input signal before applying DT, SVM, and CNN, in order to compensate the unmatched data acquisition environment. The gain factor plays the role similar to the adjustable threshold; a large gain factor is equivalent to a small threshold, and vice versa. Therefore, we can obtain an ROC curve by using various gain factors. The bias of k-means clustering also serves to make an ROC curve.

Figure 8 shows the ROC curves obtained by k-menas, DT, SVM, and CNNs with various number of intermediate layers. The results of threshold-based classifier, which is exactly the same as η = 0.07 of Fig. 4, was added for comparison. It is clear that the model-based classifiers achieved better results than the threshold-based classifier. However, unsupervised training by k-means clustering did not improve the accuracy. As for CNNs, unlike the preliminary evaluation, the shallowest network achieved the most accurate VAD results. The different tendency regarding to the number of layers can be attributed to the overfitting effect. In the preliminary evaluation, the deeper CNN can learn even small details of the training data which can contribute only under the matched condition. In the CENSREC-1-C evaluation, the shallower CNN has a more generalized model, which contributes to avoid errors related to the condition-specific features. That argument is supported by the fact that the superiority of DT over SVM is not observed in the CENSREC-1-C evaluation, because DT is more likely to overfit the training data even though it includes some pruning algorithms.

Fig. 8. ROC curves for CENSREC-1-C obtained by various classifiers. DT and SVM represent the voting results of 100 classifiers.

So far, we have found that the combination of augmented statistical noise suppression and CNN-based classifier achieved the best VAD accuracy among considered algorithms. In Fig. 9, it is demonstrated how the ROC curve shifted toward the left-bottom corner. SNS-thres, which is the copy of Q(l) of Fig. 1, is our starting point, and it was greatly improved by augmented statistical noise suppression, resulted in ASNS-thres. There was some additional improvement by introducing CNN, and the final ROC curve was plotted as ASNS-CNN. We also plotted some published results using CENSREC-1-C, such as using CRF [Reference Saito, Nankaku, Lee and Tokuda16], SKF [Reference Fujimoto and Ishizuka19], and SKF with Gaussian pruning (SKF/GP) [Reference Fujimoto, Watanabe and Nakatani31], and confirmed that the proposed algorithm outperformed them.

Fig. 9. ROC curves for CENSREC-1-C obtained by the proposed algorithms. Reference points of CRF, SKF, and SKF/GP were cited from the published papers.

Following the convention of CENEREC-1-C, the details of the VAD results obtained by ASNS-CNN are show in Fig. 10.

Fig. 10. Detail of CENSREC-1-C evaluation.

E) Computational complexity

In addition to the VAD accuracy, we confirmed the processing time and latency of the proposed algorithm. Table 3 shows the processing time for the 144 files of CENSREC-1-C dataset (6465.33 s). The noise suppression and classification programs were implemented separately on Ubuntu 16.04 running on Intel Core i7 processor (3.6 GHz) with 16 GB RAM. Although the programs were not optimized in terms of speed, the real-time factor (RTF) was 0.051, meaning that the total processing time was about 1/20 of the real time.

Table 3. Processing Time for CENSREC-1-C.

Regardless of the processing speed, the proposed algorithm cannot avoid the framing latency. Noise suppression causes a half-frame (16 ms) latency and VAD causes five half-overlapping frame latency (60 ms). The total latency up to 84 ms (including 8 ms frame boundary adjustment) is not negligible, but the speech recognition system uses similar number of successive frames as the input feature. Therefore, the latency does not matter if the proposed VAD algorithm is used for speech recognition.