2.1. Ship Detection in Optical Imaging Data

For optical ship detection, we have trained a hybrid generative-discriminative model. To remove the effect of false alarms caused by land areas, land/sea area segmentation is firstly performed based on Geographical Information System (GIS) information. Both shape features and texture features are then exploited for optical ship modelling. In our experiments, a Histogram of Gradient (HOG) feature (Dalal and Triggs, Reference Dalal and Triggs2005) is computed for shape modelling and a pyramid binary pattern feature (Sun et al., Reference Sun, Wang, Wang and El-sheimy2011) is computed for texture modelling.

The hybrid learning model involves two processes: offline training process and online detection process, as shown in Figure 2. For offline training, the Expectation Maximization (EM) algorithm is used to fit the Probabilistic Latent Semantic Analysis (PLSA) (Hofmann, Reference Hofmann2001) model. (The feature set is denoted by F= {f ₁, f ₂, …, f _M}, the target set is denoted by O= {o ₁, o ₂, …, o _N}, and the latent topic set is denoted by Z= {z ₁, z ₂, …, z _K}). After learning the model parameter P (f _i|z_k) and P(z _k|o_i^train), each training sample can be represented by a K-dimensional vector P(z|o _i^train). Then a discriminative four-category (including cloud, island, ocean wave and ship) Support Vector Machine (SVM) classification model is learned using P(|o _i^train) and its corresponding category label. During online detection, the model parameter P(f _i|z_k) from training samples is used to fit the PLSA model of testing samples. The learned SVM model is then adopted to categorise the testing samples based on P(z|o^test).

Figure 2. Ship detection in optical images.

2.2. Trajectory Clustering of ELINT Records

ELINT satellites are designed to intercept radar emissions. ELINT involves the collection and analysis of intercepted signals by other than the intended recipient. It involves the exploitation of signal “externals,” referring to the characteristics of the actual transmitted signal (including frequency of carriers and subcarriers, modulation, bandwidth, power level, etc.), beam footprint parameters, and emitter location and motion. A collection signal parameter can be used to obtain a Radio Frequency (RF) fingerprint for each emitter/emitter platform, which can then be used to locate and rapidly identify the specific emitter or emitter type in subsequent intercepts.

As ELINT acquires dense observation of target position and attribute in a short period, target detection in ELINT data is essentially a trajectory-clustering problem. We proposed a two-level clustering algorithm for target detection in ELINT data (as shown in Figure 3). Radiation signals are firstly classified by their RF fingerprints. Then, spatial-temporal constraints are applied to cluster signals with similar RF fingerprints into meaningful trajectories.

Figure 3. Trajectory clustering of ELINT measurements.

3.1. Category-level Attributes

As measurements from ELINT records and optical imagery are different, a unified definition of target attributes would benefit the fusion process. Considering that target category is of particular interest for tactical and strategic operational analysis in large-scale maritime surveillance, ship attributes are defined as the probability of belonging to a specific class.

For ELINT satellites, the carrier frequency, pulse width and repetition interval of the radiation source signal can be obtained after splitting up the intercepted signal and feature extraction. With the support of an intelligence database (which is acquired offline by manual efforts), we can identify the target category based on the radiation source. However, it should be noted that targets from the same category can carry different types of radiation source and the same type of radiation source can be carried by different target categories. Let R= {R _t|t ∈ [1, T]} and C= {C _m| m ∈ [1, M]} denote the radiation source set and target category set respectively, U _i denotes the Electronic Reconnaissance (ER) record of target i and the ER measurement can be written as:

(1)$$U = \left\{ {U_1, U_2, \cdots U_n} \right\},U_i = \left\{ {U_{i1}, U_{i2} \cdots, U_{iT}} \right\}$$

where U _it denotes the probability of target i carrying radiation type R _t. This is computed by extracting the carrier frequency, pulse width and repetition interval of the radiation signal and matching these features to the database. Based on the relationship between radiation source and target categories from intelligence database, we can get:

(2)$$V_t = \left\{ {V_{t1}, V_{t2}, \cdots V_{tM}} \right\}$$

where V _tk denotes the probability of radiation source R _t carried by target category C _k. The probability that target i belongs to target category C _k based on ELINT measurement is then computed as:

(3)$$s_{ik} = \sum\limits_{t = 1}^N {U_{it} V_{tk}} $$

For optical observations, we can perform object category recognition using various features. The aspect ratio (length/width) of different ships is a simple yet very robust discriminative feature and is used for optical attributes computation. Considering that the extraction of aspect ratio of different ships in optical imagery may suffer from measurement error, fuzzy set theory is introduced for ship category description and a modified normal distribution is used as the membership function:

(4)$$p_{i'k} = \left\{\!\!\matrix{\exp \left[ { - \displaystyle{{\left( {t_{i'} - L_{k1}} \right)^2} \over {2\sigma ^2}}} \right],\quad t_{i'} \le L_{k1} \cr 1,\quad L_{k1} \lt t_{i'} \lt L_{k2} \cr \exp \left[ { - \displaystyle{{\left( {t_{i'} - L_{k2}} \right)^2} \over {2\sigma ^2}}} \right],\quad t_{i'} \ge L_{k2} } \right.$$

where p _i′k denotes the probability that target i′ belongs to target category C _k, σ is the limit coefficient and is determined based on empirical estimate, t _i′ denotes aspect ratio of target i′, [C _k1, C _k2] is the dynamic range of C _k.

Combining category-level attributes from ELINT observations and optical imaging satellite observations, the basic probability assignment function is given as:

(5)$$Sp_{ii'} = \sum\limits_{k = 1}^M {s_{ik} p_{i'k}} $$

The basic probability assignment function of point pairs s _i↔t _i′ and s _j↔t _j′ based on category-level attributive characteristics can be computed as follows:

(6)$$Sp\left( {i,j,i',j'} \right) = Sp_{ii'} Sp_{\,jj'} $$

3.2. Topology Attributes

Motivated by the observation that ship groups performing tactical or strategic operations often have a stable topology in a short time period, we present a robust point set geometric descriptor, named Point Pair Topological Characteristics (PPTC). The computation process of the PPTC descriptor is shown in Figure 4.

Figure 4. Point set topology descriptor.

Given a point set S composed of n points, the PPTC of point s _j relative to point s _i is:

(7)$$PP_{ij} (s_i ) = \left\{ {\rho \left( {s_i, s_j} \right){\rm,} \theta \left( {s_i, s_j} \right)\vert s_i, s_j \in S} \right.\left. {;\rho \left( {s_i, s_j} \right) \in \left[ {1,p} \right];\theta \left( {s_i, s_j} \right) \in \left[ {1,q} \right]} \right\}\;$$

where m _s is the mass centre of the point set, ρ (s _i, s_j) denotes the normalized distance from point s _j to point s _i, θ(s _i, s_j) denotes the normalized angle between vector $\mathop {{\bi s}_{\bi i} {\bi s}_{\bi j}} \limits^ \to $ and vector $\mathop {{\bi s}_{\bi i} m_{\bi s}} \limits^ \to $, p and q are the quantized number of distance and angle respectively. The PPTC descriptor of a point set is formed by concatenating all PP _ij(s _i) together. The PPTC descriptor has several advantages for topology preservation. Firstly, it normalizes and quantizes the logarithm distance between point pairs. It is insensitive to measurement error such as point position deviation. Secondly, each point set has its separate coordinate system, which makes the descriptor robust to inaccurate orientation between point pairs. Thirdly, the PPTC is a global topology descriptor and it is robust to noise and local measurement errors.

Based on the PPTC descriptor, a new basic probability assignment function of point pairs s _i↔t _i′ and s _j↔t _j′ is computed as follows.

(8)$$\alpha = \exp \left( {{{ - \displaystyle{{\left\vert {\rho _{i'} \left( {t_{i'}, t_{\,j'}} \right) - \rho _i \left( {s_i, s_j} \right)} \right\vert } \over {\mathop {\max} \limits_{m,n'} \left\{ {\rho _m \left( {s_m, s_n} \right),\rho _{n'} \left( {t_{n'}, t_{m'}} \right)} \right\}}}} \Bigg/ {\sigma _\alpha ^2}}} \right)$$

(9)$$\beta = \exp \left( {{{ - \displaystyle{{\left\vert {\theta _{i'} \left( {t_{i'}, t_{j'}} \right) - \theta _i \left( {s_i, s_j} \right)} \right\vert } \over {\mathop {\max} \limits_{m,n'} \left\{ {\theta _m \left( {s_m, s_n} \right),\theta _{n'} \left( {t_{n'}, t_{m'}} \right)} \right\}}}} \Bigg/ {\sigma _\beta ^2}}} \right)$$

(10)$$Cp_{ij,i'j'} = \left( {1 - \alpha} \right)\left( {1 - \beta} \right)$$

(11)$$Cp(i,j;i\hskip1pt',j\hskip1pt') = 1/(1 + \left( {Cp_{ij,i\hskip1pt'j\hskip1pt'} + Cp_{ji;j'i'}} \right)^2 )$$

where α and β denotes the similarity of logarithm distance and angle between point pairs, σ _α and σ _β denotes covariance of distance and angle respectively, Cp _ij,i′j′ is the similarity between Pt _ij(s _i) and Pt _i′j′(t _i′), Cp(i, j;i′, j′) is the basic probability assignment function of s _i↔t _i′ and s _j↔t _j′. Small values of Cp _ij,i′j′ indicates a high level of topology similarity.

4.1. Coherent Point Set (CPS)

Given the template point set S= {s ₁, s ₂, …, s _n} from optical satellite data and the target point set T= {t ₁, t ₂, …,t _m} from ER satellite data, our goal is to fuse the observations by point set association matrix M = [M _ij]_n×m, whose element M _ij ∈ [0, 1] denotes the corresponding relationship between s _i ∈ S and t _j ∈ T. If the similarity measurement of (s _i, t_j) is C _ij, the cost matrix of matching the two point sets is:

(12)$$C(S,T,{\bi M}) = \sum\limits_{i = 1}^n {\sum\limits_{\,j = 1}^m {M_{ij} C_{ij}}} $$

Association of multi-target groups is equivalent to finding the optimized point set association probability matrix ${\bi \hat M}$ under the constraint of one-to-one mapping.

(13)$${\bi \hat M} = \arg \mathop {\min} \limits_{\bi M} C(S,T,{\bi M})$$

In order to combine ship group category-level attributes and topology attributes efficiently within a unified framework, we use the D-S evidence theory. The core idea of D-S evidence is to get the total basic probability assignment function by integrating the basic probability assignment function of each item of evidence. The synthesized basic probability assignment function can be computed as:

(14)$$C(i,j,i',j') = \displaystyle{{\sum\limits_{\alpha = (i \leftrightarrow i',j \leftrightarrow j')} {Cp(m_1, n_1 ;m_1 ^\prime, n_1 ^\prime )*Sp(m_2, n_2 ;m_2 ^\prime, n_2 ^\prime )}} \over {1 - \sum\limits_{\alpha = \Phi} {Cp(m_1, n_1 ;m_1 ^\prime, n_1 ^\prime )*Sp(m_2, n_2 ;m_2 ^\prime, n_2 ^\prime )}}} $$

where α= (m ₁↔m ₁′, n ₁↔n ₁′) ∩ (m ₂↔m ₂′, n ₂↔n ₂′). To get a global coherent result, relaxation labelling technique (Ouyang et al., Reference Ouyang, Ji and Tian2012) is used to refine the iterative process. Firstly, we obtain the initial association probability matrix M ⁽⁰⁾ = [M _ij⁽⁰⁾]_(n+1)×(m+1) by C(i, j;i^′, j^′), where M _ij⁽⁰⁾ is the initial probability of s _i↔t _j. For any point s _i^′ (i′ ≠ i), only one point t _j′ (j^′≠j) is associated with t _j. Let $\mathop {\max} \limits_{\,j' \ne j} \left[ {C(i,j;i',j')} \right]$ denote the support of s _i^′↔t _j^′ for s _i↔t _j, the initial association probability matrix M _ij⁽⁰⁾ is the average support for s _i↔t _j from all s _i^′↔t _j^′ (i′≠i, j′≠j). At the r^th iteration (r > 0), the support [M _ij^(r)]_(n+1)×(m+1) of s _i^′↔t_j^′ for s _i↔t _j is not only related to C(i, j;i′, j′) but also to M _i′j′^(r−1).

4.2. Sensitivity Analysis

We simulate a scenario of maritime surveillance of 100 × 100 kilometres, where a ship group of nine ships from four different categories [L ₁, L ₂, L ₃, L ₄] is present, including one ship from category L ₁, two ships from category L ₂, three ships from category L ₃ and three ships from category L ₄. The ground truth of ship numbers for optical imaging satellite and ELINT satellite is denoted by nP = nQ= 9. For optical imaging measurement, an image with 5000 × 5000 pixels of 20 metres resolution is generated. Nine points are randomly selected from the image as the template points. Each point has a corresponding aspect ratio k _ratio drawn from [L _k1, L _k2] of category L _k as its optical attribute. The observed aspect ratio is randomly generated from [L _k − σL_k, L_k + σL _k] where σ is the deviation error. The ELINT observations are simulated by transforming the optical point positions with a similarity transformation. The transformation parameters are 0·1 ⩽ s⩽5·0, −π ⩽ θ ⩽ π, −2000 ⩽ t _x, t_y ⩽ 2000, where s is the scale factor, θ is the rotation angle, t _x and t _y are the displacements. As the ELINT data and optical image data have different location precision, a zero mean Gaussian noise with covariance δ = f × d _minT (d _minT is the minimum distance between targets, f is the noise level factor) is added to the ER position. There are also false alarms in the ELINT observations and they are modelled by adding nQ _o = γ × nQ points to the ELINT data.

For the simulated experiments, we use the parameters setting f= 1, σ= 0·5, s= 0·9, θ = −(3/4) π, the results using different methods with nQ _o = 0 and nQ _o = 20 are shown in Figure 5 respectively. The red icons denote observation points from ELINT data and the green icons denote observation points from optical data. The geometrical forms denote target type. In both ELINT and optical data sets, point category information is given manually.

Figure 5. Association results using simulated data. (a) nQ _o = 0. (b) nQ _o = 20. The left column shows association results using topology features, the middle column shows association results using category-level attributes, and the right column shows association results of the proposed method.

To test the robustness of the proposed algorithm to the location noise, the target size deviation and the outliers, we ran two groups of experiments on the simulated data with different degrees of noise and different levels of outlier ratio. We ran 200 experiments for each group. Figure 6 shows the correct matching rate varying with the noise level factor f and the target size deviation error σ. The results demonstrate the robustness of our proposed method.

Figure 6. Robustness to measurement error. (a) The correct matching rate varying with f. (b) The correct matching rate varying with σ.

5.1. Optical Ship Detection Results

We collected 316 optical images from CBERS and SPOT 4 satellites. The resolution ranges from ten to five metres, some typical samples of test images are shown in Figure 7. 1653 candidate ship region images were obtained after land/sea segmentation and candidate ship area detection. In our experiments, the candidate ship region images have been partitioned into two randomly selected halves: one half forms the training set and the other forms the testing set. The training set consists of 495 randomly selected images. The rest of the images in the data forms the testing set for separate performance evaluation. The optical ship model learnt from the training data is used to classify images in the testing set into either ships or background.

Figure 7. Several typical samples of testing images.

To measure the performance of optical ship detection, the following parameters are used:

(15)$$Recall = \vert N_C\vert\hskip-5pt\bigg/\hskip-5pt\vert N_G\vert, Precision = \vert N_C\vert\hskip-5pt\bigg/\hskip-5pt\vert N_D\vert}$}}$$

where N _c, N_G, N_D represents the number of correctly detected, ground truth and actual detected ships, respectively.

We compare our method with two state-of-the-art methods including Principal Component Analysis and K-Nearest Neighbour (PCA+KNN) and Feature+SVM (Zhu et al., Reference Zhu, Zhou, Wang and Guo2010). The PCA+KNN method is selected as it is the baseline method for optical ship category classification and the Feature+SVM method is selected as it is the most successful paradigm in the literature. PCA+KNN method firstly performs principal component analysis on the extracted features and then classifies candidate ship images using KNN (In our experiments K= 7). Feature+SVM method directly inputs the extracted features to SVM classifier. We have carried out two groups of experiments. In the first group, candidate ship images are classified into ships and background (two-class strategy). In the second group, the background category is further classified into cloud, island, ocean wave and background (multi-class strategy). The results are reported in Table 1 and Table 2 respectively.

Table 1. Statistical result of different methods using two-class strategy.

Table 2. Statistical result of different methods using multi-class strategy.

5.2. ELINT Trajectory Clustering Results

To verify the effectiveness of our trajectory clustering algorithm in ELINT records, our two-level clustering algorithm has been compared to clustering methods based on logical rules and Hough transform using two sets of recorded data. The results are shown in Figure 8 and Figure 9. It can be seen that our algorithm recovers the most meaningful trajectories.

Figure 8. (a) ELINT Records. (b) Trajectory clustering results of our algorithm. (c) Trajectory clustering results based on logical rules. (d) Trajectory clustering results based on Hough Transform.

Figure 9. (a) ELINT Records. (b) Trajectory clustering results of our algorithm. (c) Trajectory clustering results based on logical rules. (d) Trajectory clustering results based on Hough Transform.

5.3. Association Results using Recorded Data

To test the proposed algorithm in real-world applications, ships detected in the SPOT 4 optical remote sensing image (20 m resolution) forms the template point set, the location of the radiation source from the ELINT sensor observation forms the target point set. The template point set has seven members (as shown in the left column of Figure 10 (a)) while the target point set has 17 members (as shown in the right column of Figure 10 (a)). The template points and target points are depicted by different point markers.

Figure 10. Association results using recorded data. (a) Association results. (b) The association matrix.

Both shape features and texture features are exploited for optical ship modelling in the template points. Radiation signals in the target point set are classified by their RF fingerprints. For category-level attribute characterization, ships have been classified into four categories, namely aircraft carrier, cruiser, frigate and depot ship. The topology attribute descriptor is computed from the relative position of different ships in the group.

The association results using both category-level attributes and topology attributes under the framework of CPS are shown in Figure 10. Figure 10 (a) gives the visualized demonstration while Figure 10 (b) shows the numerical values of the association matrix. As a time delay exists in the optical image and the ELINT record and the ship groups in the template set and target set are not the same, it can be seen from Figure 10(a) that the CPS are robust to topology distortions. ELINT records usually have more false detections and thus generate lots of outliers in the target set. The association matrix in Figure 10(b) demonstrates our method can obtain the correct association under the presence of multiple outliers. It is suitable for real-world application scenarios.