Maritime anomaly detection is a key technique in intelligent vessel traffic surveillance systems and implementation of maritime situational awareness. In this paper, we propose a method which combines vessel trajectory clustering and Naïve Bayes classifier to detect anomalous vessel behaviour in the maritime surveillance system. A similarity measurement between vessel trajectories is designed based on the spatial and directional characteristics of Automatic Identification System (AIS) data, then the method of hierarchical and k-medoids clustering are applied to model and learn the typical vessel sailing pattern within harbour waters. The Naïve Bayes classifier of vessel behaviour is built to classify and detect anomalous vessel behaviour. The proposed method has been tested and validated on the vessel trajectories from AIS data within the waters of Xiamen Bay and Chengsanjiao, China. The results indicate that the proposed method is effective and helpful, thus enhancing maritime situational awareness in coastal waters.