The unmanned aerial vehicle (UAV) system for composite vertical take-off and landing (VTOL) is a complex, highly coupled, and nonlinear system which is sensitive to external disturbances and model uncertainties. The composite VTOL UAV system consists of a multi-rotor section and a fixed-wing section. To improve observation accuracy, the compensation function observer (CFO) uses a new structure that includes velocity information. The CFO is utilised to estimate the uncertainty and the external disturbances of the system model, which performs superior estimation accuracy compared to the extended state observer (ESO). In the modeling process of the VTOL UAV, the aerodynamic moment is calculated by means of the cross-product operation of force and force arm, which solves the problem of over-reliance on aerodynamic parameters in the traditional modeling approach. The controlled object is refined by CFO, and model compensation control (MCC) is used to realise the velocity and attitude control of the composite VTOL. The numerical simulation of MATLAB/Simulink and hardware-in-loop simulation (HIL) of Rflysim were implemented, and which were used to compare the MCC, active disturbance rejection control (ADRC), and proportion integration differentiation (PID). The simulation results confirm the superiority of MCC in controlling composite VTOL UAVs in terms of anti-disturbance and tracking speed.